Avoid 2:00 and 3:00 am cron jobs (2013)

This shouldn’t be hard to deal with if the timestamp is always serialized with the offset: I’m much more picky about always persisting the offset than about always persisting UTC

This can also be expressed as general advice.

Whenever you are unsure whether to use a clever solution or follow the globally accepted standard in your work as a DevOps or Software engineer, always choose the standard.

Nowadays I'd agree with you, UTC is probably the best bet. But back then, it wasn't.

I can’t quantify how much time my team wasted in diagnosing production glitches on checking the wrong time offsets but it was substantial. One of our systems wasn’t using UTC, and given enough time the fact that Slack wasn’t using it either does become an issue. When an outage transitions to All Hands on Deck everyone needs to get caught up to what’s going on preferably under their own power so you don’t suffer the Adding Resources to a Late Project problem.

So that first alert that came in ?? minutes ago you need to align with the telemetry and logs in order to see what the servers were doing right before everything went to shit.

While I agree on this particular instance, there are two types of things future engineers have to clean up after: Their predecessors thinking too small (and picking the easy route) or too big (and adding needless complexity).

One is not necessarily and in all instances less of a mess to clean up behind than the other.

If you use <input type=time>, the browser uses locale or user regional preferences… even if the app is in an application domain (e.g. medicine, military, science) where 24h is preferred even in 12h-predominant locales. There is no way for the app to say to the browser “this time field should be 24h in all locales”, the only option is to build a custom time field

I asked the HTML spec editors to fix this, but thus far they haven’t: https://github.com/whatwg/html/issues/6698

That is the correct and only sane way to determine date format, timezone is not the same as formatting preference. But yeah it sucks. Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.

In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages"

Just a strong hint your fellow techdorks were being at work. Oh so clever, aren't they, always?

> Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.

That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.

Does that mean you ran PST during daylight savings time and half the year your logs were “off” by an hour? Or did you actually run Pacific Time and deal with the clock changes twice a year?

What about that one hour in the day when the date isn't the same?

I'd rather the logs be consistent, and not rely on every single person who ever looks at the system logs make sure to use the correct tool the correct way.

There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.

In my experience, the hard part is getting everybody else to do that. And then also getting them to actually include the timezone in their communication with you.

> Computers can do things. Build a UX [...]

That sounds like a job for a human (at least the review part), not only a computer.

The problem is a lot of times when you are looking at logs you are already very far off the happy path of things you look at often and want to invest resources into displaying well.

Actually, back then, 18 years ago, most people expected your servers to be in Pacific or Eastern time, depends on where your company was headquartered, because none of us really had global technical workforces back then. We all pretty much worked in one office and used the local time zone, because often our servers were in the building with us or in a datacenter nearby.

Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers.

Making blanket statements like that without considering the context of the time is usually not a good idea. ;)

Can you please make your substantive points without swipes? This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

It was probably the smartest option at the time, given the context. As long as it was documented properly, no one on such a small team would have been surprised. Spend more than the 30 minutes you've spent here on HN so far, and you may learn a thing or two about what is "smart", and how that is inextricably linked to the given situation.

I believe it was an anecdote and not a recommendation.

Sounds like it was smart at the time, and then eventually outgrew the solution.

If you're like 5 people, each with a list of TODOs that doesn't fit on one screen, it's pretty smart to just do something quick and good enough, then move on, revisit it in the future.

I have a piece of paper on my desk, each side has the time zone, utc offset, and date when DST will change. Twice a year I flip over the paper.

Right now it reads "EDT, UTC-4, until: Sun, Nov 2"

Going to add a clock next to it now, that's a great idea

Analog clock is a great idea. I just use the Windows multiple timezone clocks feature, but I can see the usefulness of being able to glance at UTC.

I do the same! I am in PDT and keep a UTC and EDT clock (most of my team is east coast)

I really love this idea!

> "Our customers aren't in UTC" is often a compelling reason.

This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think?

Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.

Having tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea.

Better to localize times for users on the client-side based on their local settings, IMHO

For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.

That does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.

Always set the servers to UTC. Convert the time to local time for display, but store in UTC.

The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.

Hard math. Ah right, I think it's not till the third year of a mathematics degree that you deal with "subtracting 8"! :)

Technical debt. Please don’t.

Why not just store everything utc but add a local time to the logging along with it?

I agree with most everything except:

> If I want a report of what happened at a specific time I need that report at local time

this is hard when a particular hour can happen twice in a day right?

You can tell the job when to run in UTC time and change that time depending on what the current DST rules are.

But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all.

Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."

In the United States, we go from 1:59am to 3:00am in March, and from 1:59am to 1:00am in November for our time change.

In the UK we move forward at 1am and they go backward at 2am. Doing it at midnight adds the extra complexity that now the day is different. Doing it in the early morning doesn't change the day.

My guess is that in the US they do the same but shifted by one.

All of Europe changes from 02:00 to 03:00

Leading to my favourite web comic of all time.

https://www.monkeyuser.com/2018/going-global/

Monkey User is always entertaining.

Is there any other person other than Arthur David Olson who needed an RFC written to cover their retirement?

https://www.rfc-editor.org/rfc/rfc6557.html

Brazil not only had DST at midnight, but until 2008 they also had no standard rule for when DST would begin and end, setting the dates by decree often just a few weeks in advance.

https://www.timeanddate.com/time/time-zones-interesting.html

I wonder how on earth do you deal with a 30 or 45 minute offset in real life

I prefer to combine this with FixedRandomDelay=true. FixedRandomDelay ensures that the randomized delay is an arbitrary number up to RandomizedDelaySec, but it is deterministic per server and timer. I find this useful because this means the timer will always run at XX:12:45 on server01, always run on XX:06:23 on server02 and so on.

This combines very simple configuration, while being predictable and spreading out timers well.

With enough precision in your timestamp, you can base-10 encode arbitrary binary data and shove it in your nanos for debugging.