Question: hey! how does carbonzipper merge a timeseries if multiple backends have the same timeseries and there are missing points? will it fill in the gaps, or just pick the backend that has least gaps etc?

Asked By
tim.rice
Asked At
2017-11-29 18:22:24

Found 15 possible answers.

User Answered At Possible Answer
dgryski 2017-11-29 19:46:18 The logic could be better, but at the moment I *think* it takes the first series that comes back from the servers and fills in the missing points from the other requests. Also, https://github.com/go-graphite/carbonzipper/issues/4 and https://github.com/go-graphite/carbonzipper/issues/22 https://github.com/go-graphite/carbonzipper/blob/master/zipper/zipper.go#L168 That is in fact the use case that caused carbonzipper to be written in the first place, which started the whole zipperstack development.
tim.rice 2017-11-29 19:50:57 ok, thanks for the answer :slightly_smiling_face: thats what i suspected. am i correct in assuming that if two backends have different storage schemas then mergeing will fail?
dgryski 2017-11-29 19:56:54 storage schemas you mean retention and rollup periods? but yes; if they're not the same then zipper will skip merging
tim.rice 2017-11-29 20:00:36 yup thats what i mean :slightly_smiling_face:
dgryski 2017-11-29 20:03:24 Feel free to patch the zipper to improve the logic here :D For our original use case, the same metric with different roll-up configurations was a bug; our systems were supposed to be replicas so we tracked down those cases and fixed them as we found them
tim.rice 2017-11-29 20:07:13 ah, we're migrating to a new set of go-carbon servers using a much more "high-resolution" roll-up config, and the hope was that carbonzipper could merge them anyway while we roll out the new hosts
dgryski 2017-11-29 20:07:41 Sadly nope :( It gets tricky. Take the more-aggregated one, aggregate the higher-resolution one, and then merge? If a value has been aggregated, what do you put in? At least not at the moment. The problem really is "how do you merge timeseries when things don't match up?"
tim.rice 2017-11-29 20:09:05 yeah i understand, it was a very niche use case for us :slightly_smiling_face: no big problem
dgryski 2017-11-29 20:09:14 Or take the higher resolution one and make guesses at the individual points from the rolled-up version?
tim.rice 2017-11-29 20:11:28 well anyway, it was a long shot :wink: carbonzipper/api are still awesome! thanks for the work on them :slightly_smiling_face:
dgryski 2017-11-29 20:21:14 \o/
civil 2017-11-29 21:33:52 My answer is to interpolate them :slightly_smiling_face: But that's still a thing that must be implemented :slightly_smiling_face: AFAIR interpolation is what graphite-web do [November 29th, 2017 9:08 PM] dgryski: At least not at the moment. The problem really is "how do you merge timeseries when things don't match up?"
dnarahari 2018-01-09 18:17:42 Hi! Is the echo hanging normal? The echo hangs and no metrics on graphite echo "test.bash.stats 42 date +%s " | nc graphite.example.com 2003 I have been trying to send metrics manully
leitzler 2018-01-11 11:36:19 @dgryski / @civil would it make sense to add a feature to go-carbon (carbonserver part) that makes it possible to force a file list update? I.e. expose https://github.com/lomik/go-carbon/blob/master/carbonserver/carbonserver.go#L1750 to an endpoint or something? I have 5 minute set in the configuration, but sometimes when I restart services I would like to see the metrics right away (new files are created).
civil 2018-01-11 12:38:59 yes, why not :slightly_smiling_face:

Related Questions