Process Start Time Metric

[zackman0010]

For some of our services, we're currently using the Java Agent to open a Prometheus server and scraping the metrics using Grafana Alloy. However, we recently noticed that the OTel provided metrics do not include process_start_time_seconds like the metrics we see from Spring Boot apps. In Grafana, we use this metric for restart detection: If the metric changes, we know the service was restarted. Is there something we can enable for the Java Agent to produce a similar metric?

[steverao]

Is there something we can enable for the Java Agent to produce a similar metric?

As far as I know, there isn't a such metrics in OTel Java Agent. According to your description, could you use other metrics such as jvm.thread.count to detect restart process?

[zackman0010]

Looking at the data, it unfortunately doesn't appear that can be easily used. The JVM thread count doesn't noticeably change on restarts. I guess if there's nothing built-in, I can add something into our internal extension JAR that adds a start time metric.

[trask]

@zackman0010 would something like jvm.uptime suffice?

we don't have that metric currently, but it seems like a good candidate metric based on process.uptime and RuntimeMXBean#getUptime

if not, it would be worth opening an issue in semantic convention repository on how restart detection can be handled, maybe there's a way already or if not it seems like something that would be good to address

[zackman0010]

An uptime metric would work, yes! We'd have to adjust our monitoring to check for a decrease as opposed to a change in general, but that should be simple: just updating the changes function to resets.

To cover our need in the meantime, though, I've added this to our extension JAR to get the start time metric:

@AutoService(InstrumentationModule.class)
public class CSStartTimeInstrumentation extends InstrumentationModule {
    
    static {
        GlobalOpenTelemetry
                .getMeter("cs-start-time")
                .gaugeBuilder("process.start.time")
                .setUnit("seconds")
                .setDescription("Start time of the process in seconds")
                .build()
                .set(System.currentTimeMillis() / 1000d);
    }

    public CSStartTimeInstrumentation() {
        super("cs-start-time");
    }

    @Override
    public List<TypeInstrumentation> typeInstrumentations() {
        return Collections.emptyList();
    }
}
 

It's a little naive, but it works. It also has the bonus of having the same name as the Prometheus metric, so it'll be picked up automatically by our existing restart detection alert.

[trask]

opened open-telemetry/semantic-conventions#2022