On 2005-02-10, Scott Turner wrote:

On 2005 February 10 Thursday 07:19, Simon Marlow wrote:

...

On 10 February 2005 12:12, Marcin 'Qrczak' Kowalczyk wrote:

...

And the simple fact that obtaining TAI from systems in use requires periodic updates of the leap second table causes other difficulties.

This is a vitally important bit of rationale which has emerged from the current discussion.  I certainly didn't fully appreciate the difficulties with basing the library on TAI before,

Another significant bit for me was the realization that TAI is no good for scheduling future events on an everyday calendar. If I schedule a new year's celebration for Jan 1, 2006, that would be represented in TAI as 1514764832.0 seconds after the epoch. If a leap second is inserted at the end of 2005, then my celebration at 1514764832.0 TAI will be premature. Even if we go to the trouble of keeping our leap second tables current, this is still a problem.

Yes, because TAI is a clock, not a calendar. -- Aaron Denney -><-

Seth Kurtzberg writes:

...

Yes, because TAI is a clock, not a calendar.

We are getting bogged down in the terminology here.

I've tried -- but am happy to try again -- to propose: clock : something that measures the passage of time, unit is seconds calendar: something that deals with the passage of time with other units -- hours, minutes, years, dates I guess one reason this is not a simple and useful distinction is that people prefer POSIX' time, where the second count is computable from the UTC calendar alone (and vice versa), and thus a POSIX "second" (a "psecond"?) is sometimes two SI seconds, and sometimes none. I think it is desirable that a clock has good accuracy, while it is less of a problem if a calendar is wrong -- new year celebrations nonwithstanding.

My problem is that, as proposed, the library will give wrong answers.

The problem is wanting a one-to-one correspondence between the clock (counting seconds) and the calendar (describing other time entities, like hours or dates). POSIX is AFAIK the only way to handle this, and it is done by changing the definition of "second". A POSIX clock (unlike an SI-seconds based one) will of course only be able to keep time correctly or calculate correct durations (as SI seconds) if has access to leap second tables. The advantage of POSIX is that you don't need the leap second table to calculate the UTC calendrical presentation of time. Well, most of the time, anyway -- it won't work *during* a leap second, I guess. If you store the time to new year as SI seconds, you will have the problem of leap seconds, even if you use UTC -- you will need to know whether 2005-12-31T23:59:60 exists or not. (Or you can go the .NET route, which just pretends it is an error.) And of course, accurate time depends on distributing leap second tables anyway, as you need to know when to slow down the POSIX clock.

Having a nanosecond granularity library that can't even manage one second accuracy doesn't seem useful to me.

We would be in good company. With all the legacy baggage, it is rather utopian to expect computers to be able to deal sensibly with leap seconds for the foreseeable future.

I can't see the logic in a library that returns incorrect answers, because to return correct answers is more difficult.

Accurate timekeeping really isn't that important for computers. -kzm -- If I haven't seen further, it is by standing in the footprints of giants

Seth Kurtzberg wrote:

Timekeeping is crucial for certain applications, which happen to be the ones I work on. [...] If you ignore the realities of time (that is, that the universe isn't much interested in our simplifications) you get your satellite in the wrong place, and you lose $100 million. [...] If it's worth doing, don't return the wrong answer. [...] The universe doesn't stop for that period when POSIX pretends that it does. [...] Given the choice of reality or POSIX, it seems to me that reality is a more sensible choice.

I honestly think you ought to sit down calmly, take a stress pill, and think things over. You want the library to do the Right Thing. Great, we all do. But it's IMPOSSIBLE. If it were difficult, we would have just done it, and none of this long discussion would have happened. What, precisely, do you want from a standard time library that Ashley Yakeley's proposal could provide and doesn't? Do you want it to interface with a GPS receiver, or connect to an Internet time server to get the current time? It doesn't make sense to include such things in an initial proposal. Once we have some basic datatypes for UTC and TAI times, then we can easily write other libraries with the appropriate functions of type IO AbsoluteTime. -- Ben

Seth Kurtzberg writes:

The difficulty, if it exists, comes from converting the absolute time into calendar time, but even that isn't really difficult. The firmware in a GPS receiver does it. So it is clearly not impossible.

It can do it because it gets the current TAI-UTC offset (strictly speaking, GPS-UTC offset, where GPS time is in a fixed relation to TAI) from the satellite, together with the timestamp. It cannot convert arbitrary future times. Nobody can, because leap seconds are only known 6 months in advance. As opposed to a GPS receiver, a Haskell program doesn't usually have access to a timer from which one can reconstruct TAI algorithmically.

Why is it unreasonable to ask the library to tell me the correct answer from, say, the current time to some time in the future?

How is the time in future specified? If it's absolute TAI seconds, then the library can indeed tell it; the program could as well compute it itself, because it's a trivial subtraction of two numbers. If it's yyyy-mm-dd hh:mm:ss, then a reasonable library can't answer this with sure when the future time is at least 6 months after a version of the Haskell implementation has been released, installed, and used to compile the program. POSIX time is indeed not suitable for exact calculations of timespans with a precision of seconds and the length of months. They are suitable for calculating in terms of minutes and larget units for arbitrarily large spans (because a day is always 86400 POSIX seconds) and for storing yyyy-mm-dd hh:mm:ss UTC timestamps in the past and in the future (except that *during* a leap second it's off by one second). They can also measure the length of a timespan shorter than 6 months with is exact or 1 second too long / too short - for most programs the rare error of one second is tolerable. OTOH TAI is suitable for exact measuring the length of timespans, and for nothing else. It's not suitable for converting timestamps to/from yyyy-mm-dd hh:mm:ss, until a system-wide automatically-updated leap second table is designed and implemented, unless you don't mind having to release, download and install a Haskell compiler and recompile all programs every 6 months. Under the same conditions it's not suitable for getting the *current* time, accurate to a second, because the system clock shows POSIX time; nor for getting the last-modified time of a file, because it's also POSIX time. -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

Seth wrote:

I want to compute the correct interval between two timestamps. I guess I not only don't see why it's impossible, I don't even see why it is difficult. The difficulty, if it exists, comes from converting the absolute time into calendar time, but even that isn't really difficult. The firmware in a GPS receiver does it. So it is clearly not impossible.

The firmware in a GPS receiver can tell you the _current_ time, in both TAI and UTC, because it is continually receiving signals from the satellite containing the pair (TAI_time,UTC_offset). If the satellite says (n,m), then it reports n as the TAI time, and n+m as the UTC time. The Haskell time library will (a) not have access to a continuous stream of information from the GPS satellites, and (b) be asked to convert TAI to UTC at times other than the current time. The above GPS receiver behaviour (i.e., implementation of the "+" function) doesn't make it any more possible for System.Time to do the Right Thing. --KW 8-)

Ketil Malde wrote:

Seth Kurtzberg writes:

...

...

Yes, because TAI is a clock, not a calendar. We are getting bogged down in the terminology here. I've tried -- but am happy to try again -- to propose:

clock : something that measures the passage of time, unit is seconds

calendar: something that deals with the passage of time with other units -- hours, minutes, years, dates

I believe this discussion is continually bogged down in terminology :-) Here is how think of it. clock: measures passage of absolute time (time difference). Best available unit is (TAI) SI seconds (ie. cesium atom oscillations) calendar: gives time *stamps* to human related notion of time. Can be used to calculate calendar time differences, ie. human notions of time difference, like days and months, and yes, *calendar* seconds (ie. not absolute SI seconds). TAI is a clock. Sometimes they give it as date, but really, it is just SI-seconds since some epoch (specified as an UTC calendar date!) UTC is a calendar. UTC is a special calendar because its normal seconds are specified by an absolute time duration: a UTC second is a SI second. The strange thing about UTC is that days (and months etc) are defined with respect to the rotation of the earth. Therefore, a day does not always have 84600 UTC seconds, but sometimes one more or less (= leap seconds). Which days have a second more or less is defined by a leap table (which is undefined for the future). This means that, like any calendar, you can calculate UTC date differences in days and months, but not in seconds (unless you happen to know the leap seconds). Furthermore, you can not use UTC seconds as time durations, as there are sometimes leap second insertions/deletions (that last 0 SI-seconds). Now what I understand about implementation: - gettimeofday() returns the number of seconds since epoch (a UTC date) What kind of seconds does it return? Well, (A) sometimes these are just SI-seconds (which means we can derive TAI from that), but sometimes (B) it returns UTC-seconds -- ie. the absolute SI-second count, plus/minus any leap seconds that have occurred. Is there anyone on this list who can give a solid overview of what "gettimeofday" returns, and if we can distinguish (A) from (B)? I guess some/most of this is already said on this list, but it seems somewhat hidden in the posts and clouded by misinformation :-) I know that libtai assumes (A) which is wrong in general. All the best, -- Daan.

On 2005-02-11, Marcin 'Qrczak' Kowalczyk wrote:

Daan Leijen writes:

...

- gettimeofday() returns the number of seconds since epoch (a UTC date)

What kind of seconds does it return? Well, (A) sometimes these are just SI-seconds (which means we can derive TAI from that), but sometimes (B) it returns UTC-seconds -- ie. the absolute SI-second count, plus/minus any leap seconds that have occurred.

Is there anyone on this list who can give a solid overview of what "gettimeofday" returns, and if we can distinguish (A) from (B)?

POSIX says it returns an UTC date, converted to the number of seconds by a given formula. The formula assumes that each day has 86400 seconds.

POSIX says that it returns what I'll call "POSIX UTC", because of that brokenness. But POSIX is not the only standard, and people will be running on machines that are run differently.

With a strict reading of the definition, after a leap second one second is repeated (since 23:59:60 gives the same counter value as 0:00:00 of the next day) and converting the time to hh:mm:ss never yields 60 as the number of seconds.

NTP client is supposed to slow down the system clock at a leap second. If programs ask for the system time during a leap second, the clock progresses minimally, just enough to make time monotonic, and continues normally when it catches up with the UTC time. I don't know how well this is actually implemented.

If a Linux time zone is set to something beginning with "right", then the system clock is assumed to count SI seconds since the UTC epoch (i.e. since 1970-01-01 00:00:10 TAI, even though TAI has been established in 1972) and leap seconds are adjusted together with the time zone. That is, gmtime() returns TAI, and localtime() returns localtime which is relative to UTC. If a standard NTP client is used, I guess it will set the time wrong, too early by 22 seconds (today).

Correct, for now. Newer NTP standards distribute TAI as well as UTC, and some clients will start using that real soon now. I think ideally we would have two interfaces, one of which does its best guess at POSIX UTC and one which does its best guess at TAI, because a best guess at UTC converted to TAI (or vice-versa) has more paths for this inaccuracy to creep in.

I think the only way to distinguish these cases is to call gmtime() and localtime() on the current time, and check whether the difference is a nice round number of minutes which is reasonable for a time zone difference, or it ends with a strange number of seconds which suggests that it includes leap seconds.

Referenced above, okay, we "know" the clock is in TAI, but since our standard interface is UTC, converting it now requires that leapsecond table. If we're right at the edge of an update, we can be two seconds off!

This will work until the UTC-TAI offset reaches the smallest number which is a reasonable "fractional" part of a time zone offset.

15 minutes, likely. Practically forever. Even one minute buys us a really long time. I do think this is clever, though I'd really like for our guess at TAI or POSIX UTC to be exported. -- Aaron Denney -><-

In article <200502111713.47693.p.turner@computer.org>, Scott Turner wrote:

So during a leap second, the DiffTime field returned by getCurrentTime should be between 86,400,000,000,000,000 and 86,400,999,999,999,999. That is, it should indicate an hour, minute, second of 23:59:60. With current OSes this will not happen because they fudge the leap second.

Yes, and this is an important point that I should have mentioned before. Even with the UTCTime type, which can represent leap-seconds correctly, getCurrentTime is still going to be wrong during them, because the underlying C call ("gettimeofday") returns broken POSIX time.

When the improvements to NTP are available, true UTC can be supported by getCurrentTime.

It's not just NTP, there would have to be a new function in libc. -- Ashley Yakeley, Seattle WA

Marcin 'Qrczak' Kowalczyk wrote:

Daan Leijen writes:

...

- gettimeofday() returns the number of seconds since epoch (a UTC date)

What kind of seconds does it return? Well, (A) sometimes these are just SI-seconds (which means we can derive TAI from that), but sometimes (B) it returns UTC-seconds -- ie. the absolute SI-second count, plus/minus any leap seconds that have occurred.

Is there anyone on this list who can give a solid overview of what "gettimeofday" returns, and if we can distinguish (A) from (B)?

POSIX says it returns an UTC date, converted to the number of seconds by a given formula. The formula assumes that each day has 86400 seconds.

With a strict reading of the definition, after a leap second one second is repeated (since 23:59:60 gives the same counter value as 0:00:00 of the next day) and converting the time to hh:mm:ss never yields 60 as the number of seconds.

NTP client is supposed to slow down the system clock at a leap second. If programs ask for the system time during a leap second, the clock progresses minimally, just enough to make time monotonic, and continues normally when it catches up with the UTC time. I don't know how well this is actually implemented.

If a Linux time zone is set to something beginning with "right", then the system clock is assumed to count SI seconds since the UTC epoch (i.e. since 1970-01-01 00:00:10 TAI, even though TAI has been established in 1972) and leap seconds are adjusted together with the time zone. That is, gmtime() returns TAI, and localtime() returns localtime which is relative to UTC. If a standard NTP client is used, I guess it will set the time wrong, too early by 22 seconds (today).

I think the only way to distinguish these cases is to call gmtime() and localtime() on the current time, and check whether the difference is a nice round number of minutes which is reasonable for a time zone difference, or it ends with a strange number of seconds which suggests that it includes leap seconds. This will work until the UTC-TAI offset reaches the smallest number which is a reasonable "fractional" part of a time zone offset.

This scheme of detection breaks when a time zone offset itself is a strange number. For example glibc timezone tables include apparent noon times for Riyadh in 1987-1989 (a different time each day) - it seems they used that in Saudi Arabia then.

The time was reset each day, I believe at sundown, but that's a somewhat hazy memory. It was definitely reset each day based on some observed occurance (as in sun up, sun down, etc.) I seem to remember that the time was reset each day at midnight, but I can't remember why that would have made any sense. In any event, you are correct, there is nothing equivalent to a time zone for that situation. I think it is fair to say they weren't terribly concerned about leap seconds either. :)

On 2005-02-12, David Menendez wrote:

Daan Leijen writes:

...

clock: measures passage of absolute time (time difference). Best available unit is (TAI) SI seconds (ie. cesium atom oscillations)

calendar: gives time *stamps* to human related notion of time. Can be used to calculate calendar time differences, ie. human notions of time difference, like days and months, and yes, *calendar* seconds (ie. not absolute SI seconds).

TAI is a clock. Sometimes they give it as date, but really, it is just SI-seconds since some epoch (specified as an UTC calendar date!)

This could be correct, but my understanding is that TAI and UTC are both calendars that are based on the same clock. The difference is that the UTC clock/calendar conversion requires a leap-second table and the TAI conversion does not.

It's one way of looking at it, but not the most insightful, IMO. The "dates" that TAI provides need bear no relationship to the times and days we're comftorable with.

The problem is that NTP is based on a different clock

True.

where some seconds are longer than others.

No. NTP time is a straight count of SI seconds, and includes information about whether there is a leap second in the current day, but _drifts_ the epoch against which it is defined by the number of leap seconds that have passed. rfc1305> The NTP timescale is based on the UTC timescale, but not rfc1305> necessarily always coincident with it. At 0h on 1 January rfc1305> 1972 (MJD 41,317.0), the first tick of the UTC Era, the NTP rfc1305> clock was set to 2,272,060,800, representing the number of rfc1305> standard seconds since 0h on 1 January 1900 (MJD 15,020.0). The rfc1305> insertion of leap seconds in UTC and subsequently into NTP rfc1305> does not affect the UTC or NTP oscillator, only the conversion rfc1305> to conventional civil UTC time. However, since the only rfc1305> institutional memory available to NTP are the UTC timecode rfc1305> broadcast services, the NTP timescale is in effect reset to rfc1305> UTC as each timecode is received. Thus, when a leap second rfc1305> is inserted in UTC and subsequently in NTP, knowledge of all rfc1305> previous leap seconds is lost. Another way to describe this rfc1305> is to say there are as many NTP timescales as historic leap rfc1305> seconds. In effect, a new timescale is established after rfc1305> each new leap second. Thus, all previous leap seconds, not rfc1305> to mention the apparent origin of the timescale itself, lurch rfc1305> backward one second as each new timescale is established. If rfc1305> a clock synchronized to NTP in 1990 was used to establish the rfc1305> UTC epoch of an event that occurred in early 1972 without rfc1305> correction, the event would appear fifteen seconds late rfc1305> relative to UTC. However, NTP primary time servers resolve rfc1305> the epoch using the broadcast timecode, so that the NTP clock rfc1305> is set to the broadcast value on the current timescale. As a rfc1305> result, for the most precise determination of epoch relative rfc1305> to the historic UTC clock, the user must subtract from the rfc1305> apparent NTP epoch the offsets shown in Table 8 at the relative rfc1305> epoches shown. This is a feature of almost all present day rfc1305> time-distribution mechanisms. (note that rfc 1305 uses "epoch" differently than the standard in Unix. For NTP, epoch is the reading of some clock at some specific point in time, not the zero of a count of time-units.) NTP client _implementations_ on _POSIX systems_ slow down the local clock slightly for a few minutes before the transition. This is entirely due to POSIX UTC being unable to represent the correct time, and wanting to skip directly from 11:59:59 to 12:00:00, rather than going through 11:59:60, and deciding that slowing down the clock is better than repeating a second.

...

Now what I understand about implementation:

- gettimeofday() returns the number of seconds since epoch (a UTC date)

What kind of seconds does it return? Well, (A) sometimes these are just SI-seconds (which means we can derive TAI from that), but sometimes (B) it returns UTC-seconds -- ie. the absolute SI-second count, plus/minus any leap seconds that have occurred.

I think you mean "POSIX-seconds" instead of "UTC-seconds" here. Otherwise, (A) and (B) seem identical to me.

UTC POSIX seconds, or NTP seconds, or (UTC days/86400) + number of seconds that have passed in the current day. -- Aaron Denney -><-

On 2005-02-12, David Menendez wrote:

Aaron Denney writes:

...

It's one way of looking at it, but not the most insightful, IMO. The "dates" that TAI provides need bear no relationship to the times and days we're comftorable with.

No relationship? As I understand it, the only real difference between UTC and TAI the tradeoffs they make. TAI prioritzes constant durations for its units, and UTC prioritizes tracking the irregular rotation of the Earth. TAI agrees with the common definition that a minute is 60 seconds, and UTC approximates the common definition that a day corresponds to one rotation of the Earth. You can't do both unless you mess with the definition of a second, like UT1.

What I meant that any given datestamp would have no fixed means of conversion between the two. "Need bear". Currently they're offset at roughly half a minute. In the future this offset will be different.

The problem with drifting the epoch is that it complicates clock/calendar conversions for times in the past. That is, a function secondsFromEpoch :: CalendarTime -> ClockTime will return different values for the same input if it is evaluated at different times.

Not necessarily. You can remember where the jumps happened, and know what the epoch was for specific CalendarTimes and ClockTimes. This fails for old timestamps referencing the future relative when they were created, but those are broken anyways. This doesn't fix everything as around leap seconds there are two consecutive seconds with the same timestamp -- Aaron Denney -><-