Latest events
10-06-2024 ETH node degraded performance Due to low quality code being committed by the contributor "Holiman" to the main Ethereum repository (go-ethereum), our ETH node is experiencing instability for last days affecting orders involving ETH and ERC-20 tokens. Any orders created to receive or send these currencies may face deposit detection and payout delays till the situation is reviewed and resolved. If you don't want to experience unexpected order delays, please do not trade ETH on our platform till the further update of this notification. Ticket response times related to ETH and ERC20 currently vary from 12 to 24 hours.
UPDATE: The transaction broadcasting issue was resolved by implementing a patch to the bug that the core developers refuse to fix or admit as an issue. The Go-ethereum core development team is the most incompetent team in the industry. There is even no point to report issues to them at all. Our service has currently no issues.
07-06-2024 ETH node downtime Ethereum node crashed and corrupted its blockdata files itself. We are currently resyncing the whole blockchain which will take some hours. All affected orders (that involve ETH, USDT, DAI and USDC) will be processed once the resynchronitzation is complete.
26-05-2024 Service outage

One of our core servers had all disks full which resulted into frozen VMs serving some core APIs and cryptocurrency nodes. After a cleanup and resuming VMs, some cryptocurrency nodes became unsynced with their networks due to system time desync while VMs were suspended. We have reset system time on all VMs as well as hypervisor's system time which resolved the problem.

All pending orders (500+) that were stuck during that time were processed.

14-05-2024 Service downtime (~3 hours)

Our backend engine node server became unresponsive due to hardware failure related to the network adapter's kernel bug (related and absolutely same behaviour reported here)

10 minutes after server unresponsiveness we were eventually able to login via SSH by some attempts to see that the problem is the failing network adapter caused by IOMMU-related kernel bug triggered inside our Tor router VM.

We have assumed safe to soft-reboot the server since there were no signs of any physical server intervention. We have also checked the bootloader integrity and changed disk encryption keys after a reboot.

No disk or RAM modules were ever pulled from the server. No USB ports were triggered or unknown devices attached to the server.

Conclusion: Not a coldboot attack attempt - safe to operate. Migration to other hardware will still be performed later.

25-03-2024 - 05-04-2024 XMR RPC API degraded performance / stuck XMR payouts Since Mar 25 we are experiencing intermittent API errors coming from `monero-wallet-rpc` resulting into randomly failed payouts. We will wait for the Monero core team to investigate the issue and release a patch but there is no ETA. If your order's XMR payout is stuck, the error will be cleared within an hour and another payout attempt will be done. We have implemented a temporary workaround to automatically monitor failed `transfer` API calls that had a response { 'code' => -38, 'message' => 'no connection to daemon' } and repeat them after some minutes. (Node and RPC version 0.18.3.3)
UPDATE:
We have identified the cause after some debugging and found out that there were reports of this issue already (1, 2, 3, 4, 5) but the core team refuse to address it, suggesting enabling hugepages instead of assigning and fixing the obvious bug. We are not going to enable hugepages because 1) we can't due to internal reasons 2) the issue is a broken Monero node code and not anything else, but what devs are currently proposing is sealing a leaking boat with a duct tape. The bug is approximately 2-years old.
UPDATE:
Fixed by recompiling without libunwind (unset LIBUNWIND_FOUND). This solution was never proposed or mentioned by the core team in relevant Github issues.
08-03-2024 XMR node degraded performance The RPC client from the v0.18.1.0 release we are running had async problems blocking many RPC calls probably due to the current attack on the Monero network. Compiling and launching the RPC client from v0.18.3.1 while keeping the node at v0.18.1.0 seemed to solve the issue. The v0.18.3.1 node crashes some minutes after launch so we will keep the node at the older version till some major update, since we are currently not using pre-built binaries from the core Monero team due to some of their members having serious infosec issues, such as rogue OS (Windows) usage for development of Monero.
30-12-2023 ETH node downtime Ethereum node maintenance took 22 hours due to a bug named "holiman" that caused blockchain corruption. We had no choice but to resync the whole block data. We have also spun up a second backup node to prevent long downtimes in future.
Node status information
BTC ONLINE since Wed May 29 22:35:58 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 849561
BTC ONLINE since Wed May 29 22:35:58 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 849561
BTCLN ONLINE since Sun Jun 23 12:31:52 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 849561
DASH ONLINE since Sun Jun 2 19:56:25 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 2094637
ETH ONLINE since Wed Jun 26 11:49:32 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
(patched with this commit by gituser543 for signing transactions with external private keys)
Additional information (click to expand)
Occasionally, ETH (and ERC-20) transactions you send to us may take a bit longer to be detected (10-15 minutes). The Ethereum node software stack that is released as "stable" on the official site currently contains a lot of critical bugs. Random crashes, blockchain desync and even blockchain files corruption happen to our Geth instance at least 2 times per week, which makes this software extremely unstable for the production use. Geth core development team are extremely incompetent and refuse to admit or investigate such bugs while their universal solution is to resync the whole blockchain whenever their product brakes itself (example issue of an unfixed bug). Note that Ethereum is the only project with official software stack being broken among the cryptocurrencies we receive, thus, we count on your understanding and that someday the whole Ethereum development finally gets some competent developers or at least a node not written in insecure for production environment languages such as Go/Java/NodeJS. Things got especially worse after the PoS merge, since they now often throw an excuse accusing beacon chain software of being broken, which makes it simple for them to disclaim the responsibility. We use Lighthouse for the beacon chain and it's the only client that is stable and developed by the competent programmers. It's written in Rust, which isn't our favourite language as well due to dependency chain security risks, but that's still much better than Go or Java. We made some health monitoring scripts to determine when Geth gets into a broken state. If your transaction wasn't detected in time, it means our node was restarted some minutes ago and it's resynchronizing and your transaction will be detected once synchronization finishes.
Height: 20175660
LTC ONLINE since Sun May 26 10:58:32 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 2709748
XMR ONLINE since Thu May 16 13:10:33 2024
Software:
Updated: Thu Nov 14 20:07:26 2024
Height: 3179743