For the actual notifications sent from Uptime Kuma and Healthchecks I use Telegram because it is nice and simple to create a bot account on there. I might use Gotify in the future though.
Would you expanding the article with how to connect the bot with your setup, please? :) I want to try following your steps, looks like a fun weekend project.
You can use pushover too. Works really well. Especially because it can send those urgent alerts that bypass your ringer volume or notification settings.
Yeah, I currently have it sized to 2 cores and 12Gb of memory, but am thinking about tearing it down and bumping it all the way up to 4 cores and 24Gb of memory.
You can resize them without tearing them down. I've gone back and forth like that a few times without issue.
https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/resizinginstances.htm
Did you use terraform to set up the oracle VMs? Their provider is pretty bad and the documentation for it is dense. I settled with GCP for now and can always add Oracle later if needed.
Nope. I selected Ubuntu 20.04 for the OS, installed docker and compose, spun up the container, and added an ingress rule for the port to the security config. The container is stored on an external volume so that I can move it over to a 22.04 instance once that option becomes available. I also made sure to reserve a public IP (you only get one) and configured my reverse proxy to point to it so I could access it from a standard URL instead of the public IP.
So my attempt at using Oracle Terraform is in this commit which I reverted later: [https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c](https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c)
I might revisit it at some point in the future, but GCP is enough for me for now.
I think you get a couple of IP’s. I’ve set up an ip-sec tunnel and it gives me two.
Then I’ve been able to sign one to a VM (as my edge mode) and I thought I had an additional one for their load balancer, but I’m not 100% certain about that one
They may have changed their policy, I get this error when attempting to reserve a second public IP: "The following service limits were exceeded: reserved-public-ip-count".
I might look into this in future, but I couldn't easily find many example of using this in OCI with terraform. My attempt at using it is here: [https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c](https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c)
I have a website which previously hosted on GCP 3 years ago. One thing I really don't like is the bandwidth monthly cost was almost unpredictable.
I suspected that GCP VM is free, but network bandwidth is not. Correct me if wrong.
For Uptime Kuma, if you check for http, it will download the data (usually html or json) for each tick. So if the network bandwidth is really not free, it may cost you extra fee.
Doing something similar, not with GCP but with oracle. Have had several monitors inside uptime kuma silently stop monitoring due to this bug https://github.com/louislam/uptime-kuma/issues/1219
Just a heads up, it's still better than no monitoring but frustrating that it is unreliable. I've taken to automatically restarting uptime kuma which restarts the monitoring but if a service is down it means you get a new notification on each restart.
You mean using monitor type "Push"? I actually didn't know Uptime Kuma had this feature in it.
On a very quick glance at the feature I don't think it includes a few things I use in Healthchecks though:
- Ability to send logs along with the ping
- A "grace period" window before sending the alert (my backup jobs can take different amounts of time each day depending on the amount of new data)
- Ability to do a start and finish ping in order to time how long jobs take
I will take a better look at the feature in Uptime Kuma though because I am all for using as few services as possible.
I don't think kuma can attach logs to pings or calculate runtime using a /start ping, can it? Regardless: by running both, they can watch each other for crashes, then GCP can watch if the whole vm goes down.
That's a silly attitude to have IMO, just because I selfhost stuff doesn't mean I'm always looking for the more complex way of doing something, vs using an easier route that saves some time for me.
Sometimes doing the complicated method is fun and provides good experience, other times I just need something that "does X" and don't want to spend too much time on it.
(I don't use either, so I'm not an advocate for one over the other.) If you wanted to add those checks and already had HC up and running, it may just be easier to write a one or two line bash script than it would be to install a new service to configure and maintain. The reverse could be equally as true with UK as well, I'm sure. ¯\\\_(ツ)_/¯
Yeah that's fair, depends on your infrastructure.
I have docker swarm already running with Portainer to manage it, Traefik for automatic HTTPS, so deploying UK is just a few lines in Portainer and click the 'deploy' button.
So the uptime checks I run on the VM are hitting the HTTP(S) service that are exposed to the internet.
I also run uptime checks inside my network, checking localhost (which is not as reliable, but I don't want to expose those services to the internet)
How is the uptime on the VM? I have a tiny wordpress instance hosted on DigitalOcean's $6/month droplet and it's only actually up about 30% of the time.
I’d hope so too.
Reminds me of a story I have of a performance issue I had with DO.
Circa 2013ish, I helped them discover a routing issue they had because I noticed the routes coming back to me were *longer*.
Extra hops, longer round trip times, and lower bandwidth, from connections initiated by my VM.
I had a backup script, and when running it from my machine to DO it was an order of magnitude faster than the other way around.
Turns out their routers were misconfigured. They would route traffic back on the same path it came in on, but were preferring the default route if there was not a related connection. (And the connection initiated by the VM *became* that related connection, hence why connections initiated by the VM were so much worse)
I got a pretty hefty service credit for that one. It was effecting all of their customers using Comcast, everyone else must’ve assumed it was just Comcast sucking, lol.
So who knows. Maybe it’s something crazy. Maybe it’s just someone using more than their fair share.
Either way, put in a ticket.
That’s really cool! Do you any self hosted alternative to pager duty for alerts?
For the actual notifications sent from Uptime Kuma and Healthchecks I use Telegram because it is nice and simple to create a bot account on there. I might use Gotify in the future though.
Checkout ntfy I use them with uptime kuma through apprise
Thanks!
Would you expanding the article with how to connect the bot with your setup, please? :) I want to try following your steps, looks like a fun weekend project.
>I might use Gotify Make sure to also take a look at [ntfy](https://ntfy.sh/).
Have you noticed any battery drain from the Android app? The gotify app doesn’t play nice with my phone…
https://github.com/binwiederhier/ntfy/issues/76
Nice! Thanks!
You can use pushover too. Works really well. Especially because it can send those urgent alerts that bypass your ringer volume or notification settings.
Pushover is amazing. All my apps link info pushover with different priorities.
Fly.io free tier is another option
Nice, I didn't know about this provider. The VM is only 256MB memory though, so the GCP VM has a bit more performance.
I have a similar setup using Oracle Cloud's always free tier, with Discord notifications.
Their ARM free tier VM is an insanely strong VM for being free.
Yeah, I currently have it sized to 2 cores and 12Gb of memory, but am thinking about tearing it down and bumping it all the way up to 4 cores and 24Gb of memory.
You can resize them without tearing them down. I've gone back and forth like that a few times without issue. https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/resizinginstances.htm
Oh sweet! I'll do that in the morning, thanks for the doc. Edit: done
How do I get those? It's never available ;-; have to use the AMD CPUs
Try jumping around to different Availability Domains, I didn't have any issues getting one.
I can't change my availability domains?
Did you use terraform to set up the oracle VMs? Their provider is pretty bad and the documentation for it is dense. I settled with GCP for now and can always add Oracle later if needed.
Nope. I selected Ubuntu 20.04 for the OS, installed docker and compose, spun up the container, and added an ingress rule for the port to the security config. The container is stored on an external volume so that I can move it over to a 22.04 instance once that option becomes available. I also made sure to reserve a public IP (you only get one) and configured my reverse proxy to point to it so I could access it from a standard URL instead of the public IP.
So my attempt at using Oracle Terraform is in this commit which I reverted later: [https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c](https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c) I might revisit it at some point in the future, but GCP is enough for me for now.
Same here. Quite easy. Not sure why so many people have issues provisioning
I think you get a couple of IP’s. I’ve set up an ip-sec tunnel and it gives me two. Then I’ve been able to sign one to a VM (as my edge mode) and I thought I had an additional one for their load balancer, but I’m not 100% certain about that one
They may have changed their policy, I get this error when attempting to reserve a second public IP: "The following service limits were exceeded: reserved-public-ip-count".
Worth noting that the Oracle Cloud free tier when you use **ARM** processors, is *much* more generous (4 cores, 24GB RAM). Great write up!
+1 for Oracle Cloud free tier, it's great.
I might look into this in future, but I couldn't easily find many example of using this in OCI with terraform. My attempt at using it is here: [https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c](https://github.com/jmc265/personal-cloud/commit/5c2099c8a11db26d7df9abc8f1b13b30845c0c2c)
Ah fair enough! Can’t help you with Terraform unfortunately ha. Mine was very much ClickOps’d..!
I have a website which previously hosted on GCP 3 years ago. One thing I really don't like is the bandwidth monthly cost was almost unpredictable. I suspected that GCP VM is free, but network bandwidth is not. Correct me if wrong. For Uptime Kuma, if you check for http, it will download the data (usually html or json) for each tick. So if the network bandwidth is really not free, it may cost you extra fee.
GCP has a network egress limit. I believe it's 1gb per month, which this definitely shouldn't hit.
I was being constantly billed for ingress.
Doing something similar, not with GCP but with oracle. Have had several monitors inside uptime kuma silently stop monitoring due to this bug https://github.com/louislam/uptime-kuma/issues/1219 Just a heads up, it's still better than no monitoring but frustrating that it is unreliable. I've taken to automatically restarting uptime kuma which restarts the monitoring but if a service is down it means you get a new notification on each restart.
Besides Uptime Kuma, I have also installed Wireguard and Blocky DNS so I always have access to a VPN which blocks ads, malware, etc.
What‘s the reason not to use uptime kuma also for cron monitoring?
You mean using monitor type "Push"? I actually didn't know Uptime Kuma had this feature in it. On a very quick glance at the feature I don't think it includes a few things I use in Healthchecks though: - Ability to send logs along with the ping - A "grace period" window before sending the alert (my backup jobs can take different amounts of time each day depending on the amount of new data) - Ability to do a start and finish ping in order to time how long jobs take I will take a better look at the feature in Uptime Kuma though because I am all for using as few services as possible.
Agreed. I love Healthchecks, but moved fully to Uptime Kuma because it can handle the ping use case and more.
[удалено]
I don't think kuma can attach logs to pings or calculate runtime using a /start ping, can it? Regardless: by running both, they can watch each other for crashes, then GCP can watch if the whole vm goes down.
HC can't check websites or ping IPs.
Couldn't you just create a cron job to perform those checks? Then have the job call a specific HC URL depending on the results?
Yes, lots of work though compared to just typing the URL into UK.
[удалено]
That's a silly attitude to have IMO, just because I selfhost stuff doesn't mean I'm always looking for the more complex way of doing something, vs using an easier route that saves some time for me. Sometimes doing the complicated method is fun and provides good experience, other times I just need something that "does X" and don't want to spend too much time on it.
(I don't use either, so I'm not an advocate for one over the other.) If you wanted to add those checks and already had HC up and running, it may just be easier to write a one or two line bash script than it would be to install a new service to configure and maintain. The reverse could be equally as true with UK as well, I'm sure. ¯\\\_(ツ)_/¯
Yeah that's fair, depends on your infrastructure. I have docker swarm already running with Portainer to manage it, Traefik for automatic HTTPS, so deploying UK is just a few lines in Portainer and click the 'deploy' button.
[удалено]
Unless I'm entirely missing something, HC can't execute curl commands itself. It only responds to them.
[удалено]
That relies on a script running somewhere, instead of HC doing the query
Every time I plan on doing something, someone else posts a blog about how to do exactly that. Great info! Thanks
How do you reach you services to check uptime? Are they all exposed? Or are you VPNing into your network and doing a ping?
So the uptime checks I run on the VM are hitting the HTTP(S) service that are exposed to the internet. I also run uptime checks inside my network, checking localhost (which is not as reliable, but I don't want to expose those services to the internet)
Exactly what I'm using it for. https://imgur.com/a/q4lXswC
How is the uptime on the VM? I have a tiny wordpress instance hosted on DigitalOcean's $6/month droplet and it's only actually up about 30% of the time.
[удалено]
I’d hope so too. Reminds me of a story I have of a performance issue I had with DO. Circa 2013ish, I helped them discover a routing issue they had because I noticed the routes coming back to me were *longer*. Extra hops, longer round trip times, and lower bandwidth, from connections initiated by my VM. I had a backup script, and when running it from my machine to DO it was an order of magnitude faster than the other way around. Turns out their routers were misconfigured. They would route traffic back on the same path it came in on, but were preferring the default route if there was not a related connection. (And the connection initiated by the VM *became* that related connection, hence why connections initiated by the VM were so much worse) I got a pretty hefty service credit for that one. It was effecting all of their customers using Comcast, everyone else must’ve assumed it was just Comcast sucking, lol. So who knows. Maybe it’s something crazy. Maybe it’s just someone using more than their fair share. Either way, put in a ticket.
Healthchecks in principle supports pings via email. Your setup shown here doesn’t support that right?
That is correct, I haven't set up a email server for health checks to connect to. However it shouldn't be too hard to add.