Aug 13, 2019A Response to the Article: PIVX and 200+ PoS Chains Currently Vulnerable; Chains Already Under Attack

Before launching into the article itself, this statement from PIVX Developers:

We (PIVX Developers) were made aware of some suspicious behavior on the PIVX network last week (The week of August 5th) and have since been investigating the claim that our proof-of-stake algorithm is under attack.

At this point, we can confirm that the behavior is in fact NOT a resurgence of the “Fake Stake” attack from earlier this year, as the article claims.

PIVX users funds/PIVX are NOT at risk.

The network’s stability or chain trust has NOT been compromised. 

Investigations are still underway as to the aberrant behavior, and a more detailed findings report will be issued once investigations are concluded.

We are in communication with other projects and will seek to include as many as possible as our investigations progress and report to them how to properly fix any problem(s).

Of note: the author of the article (Mr. Yoon) was informed of the investigation by PIVX, but chose to jump to false conclusions instead of wait[ing] for the proper response.

Whenever people make claims and/or statements into a public space, it doesn’t matter if you are in crypto or not, any business, corporation, DAO, or movement, has to ascertain how to best address the issue, not just from the codebase side, but also the public statements and “marketing” side of things.  What we’ve observed (again, crypto realms or not) is often those who scream the loudest (or have the most salacious headlines) tend to get the immediate limelight, while the real work being done and truth of the matter is sometimes lost because folks are just diligently working away. The balance of spending time/energy on rebutting every piece of media or attack is what I would dare say MANY endeavors go through and struggle with.  I mean, let’s face it…it’s possible to absolutely BURY a project by pushing out negative media and optics, despite the very possible reality that the project is actually decent and is producing something good. Case and point – how many companies deploy grey/dark marketing tactics against competitors. If it wasn’t “viable” as a means to attain some internal goal, they wouldn’t do this.

So projects/endeavors have to figure out what to respond to, when, and how.  Now, inside of the crypto realm, there are a whole different set of rules and standards.  Let’s face it, one of the first rules many of us learned (back in the era of cryptsy/polonix/mt. Gox/btc-e trollboxes) is: 

Don’t feed the trolls.

Why? Because that’s what a troll wants.  Their “victory” comes by gaining energy as you expend yours trying to correct their FUD.  Now you’ve spent time there, instead of on actual productive tasks (like coding, bug squashing, developing, etc).  Notice I said elements specific to what developers do.

That’s one of the other elements in crypto that’s unique – developers (especially those working on the core codebase) are now the ones REALLY being called to the table to respond.  Someone can make a flippant statement like “Your code sucks” and yes, of course, that’s probably not worth a lick of a response. But then, people can piece together a narrative based on various bits of information which gets picked up by others, and suddenly there is a maelstrom online based on misinformation, and you have to figure out is it worth responding vs continuing to develop, build, and deploy.  

So put yourself in a developer’s shoes.  You are tasked with maintaining, developing, and building the core technology for this new frontier.  You are pinged 100x’s a day from scammers, bots, trolls, and the like with messages including “Hey, I need your help” or “How do I fork PIVX” or “there is an error, I need help” or “your codebase sucks” or “you have a bug…click here”.  In almost NO other sector are those who are working on this level of the codebase that “accessible” to the general public. 

THEN…on top of that, part of (good developers) role is to provide the security and safety of the codebase.  OPSec (Operational Security). Imagine a hypothetical scenario where a potential vulnerability is found. The developers (again, good ones) begin the process of ascertaining the BEST course of action.  Do you make a knee jerk reaction, yanking some code or slapping some quick fix patch in? Often this course of action results in a massive catastrophe later (and we’ve observed a number of projects that have done this).  Do you immediately tell the world (and thus open up a larger potential attack vector) before you have finalized the solution? These and probably a thousand other scenarios are what developers (I imagine) face on a daily or at least often basis.  And probably 99% of the time, you never hear about it.

Often – whenever a potential bug or exploit is found, developers will work amongst themselves (even cross projects) to come up with solutions, before ANY knowledge hits the public sector.  This is nothing new to bug reporting, where individuals submit potential vulnerabilities and exploits to companies (in hopes of receiving a reward) instead of just posting them online without any validation and confirmation that there is indeed an issue.

What I’ve observed in the past from PIVX devs is they take legitimate vulnerabilities and code exploits VERY seriously.  So much so, that they will lean on the side of ensuring they validate the claim FIRST, before coding a solution SECOND. Then and only then do they go public and make a statement.  I personally value this, because if the developers had knee jerk reactions to everything, the codebase would end up in a spaghetti scrambled mess putting the community at an even bigger risk.

Case in point:  The “Fake Stake” Exploit (which is what this article referenced as the “bug”).

Earlier this year, the decentralized Systems Lab out of the University of Illinois published a study entitled “‘Fake Stake’ attacks on chain-based Proof-of-Stake cryptocurrencies”.

While many projects and members of the crypto community were shouting (even blaming PIVX for an exploit which was not exclusive to the PIVX network), PIVX was working to validate the claims themselves before rushing into figuring out a viable solution.  What transpired over the following days was that PIVX developed an entire reproduction test suite to be able to accurately identify the exploit (Which, was essential to be able to design any solution to that exploit). That suite can be seen here: https://github.com/PIVX-Project/PIVX/pull/812

After an extensive 3 week deep dive into the PIVX PoS code, the PIVX dev team issued a report that notes: “As good as PoS is, like every protocol, it has some drawbacks. One of these is how cheap it is to provide a fake block and how much information is needed to be able to properly verify it. For this reason, PIVX developed several mitigations that are part of the following PR [https://github.com/PIVX-Project/PIVX/pull/803]”. They then outlined their 5 mitigations to the PoS “Fake Stake Attack” threats. 

You can read their whole report here: https://pivx.org/fake-stake-official-pivx-report/

Another example of the diligence of the PIVX Developers when it came to reported bugs, fixes, and announcements: https://medium.com/@dev.pivx/report-wrapped-serials-attack-5f4bf7b51701

So in summary:

PIVX developers took the inquiry seriously.

PIVX developers were methodical in their approach to determining the exploit before then determining viable solutions, which were then implemented.

Now, turning to the article in hand that was posted yesterday, let’s take a look at what was said and what assumptions were made

STATEMENTS MADE BY YOON:

• First, when I [Mr.. Yoon] attempted to contact the PIVX Core Developers, they wouldn’t talk to me directly.
  • Developers get emails, messages, pings, daily.  They range from “your code sucks “ to “hello sir, how to fork PIVX” to “you have a bug”.  Again, the developers’ primary focus is on the codebase, not in responding to every email and or inquiry.  Yes, they monitor for bug reporting and other issues, however, there are a myriad of ways to go about this reporting, engage in a conversation, and then allow the developers to validate/verify any claims.
  • It also appears that Mr. Yoon decided to push his article less than 36 hours from initially attempting to contact the PIVX developers, drawing his own conclusions, and making his statements in the article.  We’re not entirely sure why he elected to do this (rather than wait for official statements), but in any event, we will address the claims, assumptions, and inferences below.

• PIVX discord member named “bubiz” began relaying messages from the core devs, which I [Mr.. Yoon] found to be very odd
  • Why continue to engage in the conversation if you found this odd?

• The (PIVX) developers were aware of the bug and that there is nothing a PIVX fork can do except wait until 4.0 was released. Yoon then goes on to say “For a bug as serious as this one, you would think that they would have issued a statement for all the PIVX forks in existence (there’s a lot). And the BitGreen team has proven them dead wrong in their statement that there is “nothing you can do but wait till 4.0?
  • There is a bit to unpack in this. 
    • First, what “bug” is which bug (as there potentially are a few that are all being identified as the same bug).
    • Second, why wait till V4.0
    • Third, Bitgreen has “solved” the issue and thus PIVX is lying
  • Working backwards:
    • Bitgreen (BITG) was able to “fix” this issue.
      • We need to clarify.  The “issue” that BITG was experiencing was something altogether different from what Yoon was attempting to show going on in PIVX. Apparently, the “issue” going on in BITG was based on presumably the BITG developers having removed a very important consensus check (a missing nTime check).  This has nothing to do with PIVX nor a possible “exploit” PIVX may have suffered;(https://github.com/bitgreen)
      • Removing this nTime check can be VERY catastrophic and can lead to hyperinflation and eventual chain death.  This is presumably why “Dennis aka “XeZZ” from BitGreen stated “This has crippled the rewards system of several chains, and BitGreen has notified of all exchanges that it is listed on to halt all deposits and withdrawals until further notice.” The BITG attack has nothing to do with PIVX or the PIVX chain specifically.  It’s possible many other forks of PIVX has/had done this as well, thus opening themselves up to that vulnerability.
      • Secondly, BITG “solution” to their specific code comment error was to stop a significant portion of the active masternodes and actively put their chain at risk by decreasing the number of coins staking.
    • Why Wait Till V4?
      • We’ll get to this in a bit.  To address why “wait”, we have to address the first point about the so-called bug/exploit (NOT the one that BITG apparently created for themselves).
    • The “bug” or exploit that Mr. Yoon seems to be attempting to address based on what Dennis aka XeZZ stated “this address only has 87 PIVX coins but minted 48 on that address alone?
      • There are some discrepancies with that Mr. Yoon seems to be basing his conclusions on.
      • There is more than 87 coins in this wallet. It’s over 11k.
        • Main Address DHagKZ4ByFgxXe3txYysxqG5x6PvcSmwQS 
        • Owner Unknown 
        • Balance 11,625.05234493 PIVX 
        • Addresses 100 
        • with non zero-balance 100 
      • Yoon then continues to point to Dennis aka XeZZ statements that “The average stakeweight on PIVX is 9K, 2.3 coins per stake.” In essence, what should have taken 100 days to mint the staking rewards, took roughly 24 hours for this exploiter.
      • This then appears to be the “exploit” or bug that Mr. Yoon is trying to point out (again this is NOT what BITG was experiencing due to removing the critical nTime check).
      • As for getting stakes quicker than what is expected – an “exploit” of Proof of Stake algorithm…individuals have been attempting to game and push their own rewards since Proof of Stake was conceived.  The continual push for a fair Proof of Stake consensus cryptocurrency is something many have been working on and attempting to deploy, PIVX included. PoS V3 is the best iteration to date of a fair Proof of Stake consensus cryptocurrency.  That being said, it’s been known that through lots of UTXO experimentation, it’s possible to “push” the rewards. Put another way, through lots of experimentation with UTXO sizes, it’s possible to optimize ones staking rewards on a network. Now, as of the writing of this piece, I’m unsure if that is what is occurring here or if there is another factor at play.
      • However, that all being said: the total emission rate of the PIVX network is NOT / has NOT been affected. 
      • Also – there is no mathematical consensus rule preventing a low value input from staking “before it’s historically averaged expected time to stake”.
    • So back to Why Wait for V4.
      • A few potential reasons. 
      • ANY changes to code must be vetted and validated to ensure network stability and security.  Tossing a quick fix into the mix is irresponsible.
      • V4 of the PIVX wallet/codebase (from my understanding) has a LOT of network improvements (which, perhaps, would minimize and mitigate any of the presumed network “exploits”.  Again, what appears to be occurring is the skewing of the consensus model and rewards towards lower-valued inputs. This is just my interpretation of what’s been presented.
      • The release of V4 is impending and thus any enforced changes of a patch now followed by another rapid enforced change of the network could actually be riskier than what could be occurring (where a low-value staker is being able to obtain more of the rewards).  Again, the total emission rate of the PIVX network is NOT affected at this point it appears.

• The “fake stake” exploit clearly has not been fixed for PIVX, so the question is, was it ever fixed.
  • This is a false statement.  The “Fake Stake” exploit was fixed (https://github.com/PIVX-Project/PIVX/pull/803)

• Insider Foul Play Involved?
  • This just appears to be inflationary FUD at best.  Mr. Yoon pieced together a lack of Developer response in < 24 hours, a false statement about PIVX not fixing the fake stake attack, a misrepresentation of issues caused by BITG (their own code edit) as a PIVX network error, and a coincidental decrease in the observed aberrant behavior around the time that Mr. Yoon brought up these issues into the PIVX discord as a way to say “PIVX Developers can’t be trusted.”
  • All of these factors then Mr. Yoon attempts to lead the reader that the PIVX developers are somehow “involved” – in what we can only surmise as an attempt to cast FUD and doubts onto the PIVX developers (and project as a whole).
  • As for the “manner” in which one of the PIVX developers responded to him – I’ll say this: To suggest that him or any other core developer is deliberately exploiting the chain that they work tirelessly on securing, so that they can exploit what equates to a few thousand dollars, when they are paid more than that, is possibly libel.
    • Furszy is a superstar developer. We all have our strengths. Just because he came across as harsh, does not mean the data and what he said is wrong. 
    • PIVX Devs do not work for Bitgreen or for PIVX forks.
    • Expecting a full and documented response to a question within 36hrs is funny. This is open-source, not free labor.

• PIVX is responsible for every other project that uses its codebase and has forked from PIVX
  • That’s akin to saying that if you downloaded a free copy of adobe illustrator, then made edits (or never made edits actually), and then after a period of time, start banging on the door of Adobe saying “hey, why didn’t you fix my software?  I made changes to it….it’s not working, so now you fix it.” Adobe would look at you like “really mate?”.
  • In reality, PIVX is under no obligation to produce or work on behalf of any other project but PIVX.  From an endeavor standpoint (and for the community of PIVX), this is what you WANT from your developers.  To be focused on the project. To be ever updating the code. Why? Because after all, PIVX is open source. Projects are free to use its codebase (or not).  Projects are free to update their codebase as PIVX updates it’s own. Many members of the PIVX community actively work with PIVX clones and forks as well (on their own).  However, to say that PIVX is responsible for the 700+ other projects that have on their own elected to use PIVX’s codebase? That’s not taking responsibility for one’s own work.
  • Additionally, the fact that Mr. Yoon wrote this article (presumably coming to the rescue of all PIVX forks and PoS networks), and went public to expose a possible frailty in not only PIVX but ALL forks (by the authors claims) because he didn’t receive a response from core devs shows the authors disregard for ALL other PoS projects. The ethical and wise route would be to align a bunch of the developers and projects FIRST, to ascertain what (if any) the issue, behaviors, and possible exploit and or bug could be, BEFORE going public.  Otherwise, you are potentially just opening the door for attackers to further take down networks

In summary:

1. PIVX users’ funds/PIVX are NOT at risk.
2. PIVX network’s stability or chain trust has NOT been compromised. 
3. PIVX solved the “fake stake” exploit back in February.
4. There is aberrant behavior occurring on the PIVX network where low stake values are receiving more of the rewards (which is not a fair reward consensus model). This behavior is NOT affecting the coin emission or reward emission of the PIVX network.  
5. The aberrant behavior is NOT a resurgence of the “Fake Stake” attack from earlier this year, as the article falsely claims.
6. BITG did NOT fix the issue of fake stake or the behavior being observed on the PIVX network.
7. BITG created their own network issues and vulnerabilities when they removed a very important consensus check from their own code.
8. The PIVX Discord is not a “dev only” discord.  It’s a community discord, in which many of the PIVX developers (and many PIVX clone/fork developers) reside.

CONCLUDING THOUGHTS ABOUT ALL OF THIS

There are a LOT of factors at play here.

PIVX is one of the most cloned/forked projects in crypto (I think there are over 700 copies in the wild).  On top of that, there are an unknown amount of Forks of the PIVX Forks. This means that at any given point, projects out there are using a wide, wide range of the PIVX codebase.  Now, by “using” I mean, were copied and put into a new repo. After that exact moment, unless someone is actively working on that repo, they begin to fall behind from the most up-to-date codebase.  A few members of the PIVX community work with these projects and have tried to align them to work together, helping one another keep their codebases up to date. These folks also tend to stand as a bridge between the PIVX developers/codebase and these other projects.  Why? Well, let’s face it – there are 700+ clones of PIVX. We want the PIVX devs to be able to shine and excel in their work, which is focused on PIVX. This is just smart. Not just for the community of PIVX, but for the greater crypto ecosystem.  

If the PIVX developers are consistently working on “solving” the issues of these 700+ clones, then the core/main codebase becomes neglected, and thus “all” are affected.  As the core PIVX developers improve, fix, and expand the PIVX codebase, every other single clone has the incredible opportunity to benefit from this work as well. Thus, the beauty of open-sourced work.  

Yes, I can imagine, as a smaller team with maybe only a singular dev (if that), it can be frustrating to have to be waiting on PIVX to push updates or potential solutions.  However, this is the nature of the industry and ecosystem in which many willingly joined. PIVX developers are PIVX developers primarily…and we’re thankful for that.

PIVX is also in a stronger position than a lot of other Proof of Stake coins.  This isn’t a judgment, as much as it is to say, the network distribution of stakers (and coins) – combined with the current monetary value of a singular PIV, makes it more resilient and less susceptible to attacks than less secure, less distributed, less costly to attack – projects.  It’s the nature of the beast … which, is also, why many in PIVX do encourage the PIVX-forks and clones to collaborate and work together, sharing resources even.

As for how Mr. Yoon was addressed by one of the developers – I cannot speak on behalf of the developer for the manner in which he replied.  I tend to be someone who can see multiple sides at the same time. Of course, I have a bias to PIVX (and relationship with many of the community members and developers).  I can only imagine that as a developer in PIVX, having weathered the past year, the numerous code updates, etc, on top of having 700+ clones asking for advice, support, etc, on top of wanting to just code and not “deal” with the public enquires 24/7…that at some point you just get fed up with the amount of noise and you fire back a quick retort.  This becomes even more frustrating when that response is used to insinuate that you as a developer are hiding something. I’ve worked with dozens of developers. Most developers don’t communicate in the same ways many of us non-developers communicate. Reading into that and making meaning of that rarely serves anyone, let alone the PIVX community nor its clones (as Mr. Yoon tends to indicate he was looking out for).  

I personally haven’t talked with Mr. Yoon and I certainly don’t wish him any harm or ill will.  I don’t know his modus operandi for writing the article. That said, I (as a member of the PIVX community) felt compelled to write the above article to address the statements and assumptions by Mr. Yoon, and hopefully provide the other side of the coin to the world.  Crypto development is probably one of the hardest endeavors in the tech realm right now. Genuine, bona fide, cutting edge stuff (the stuff that PIVX does). I was saddened to see the conclusions Mr. Yoon drew on his own rather than waiting/dialoguing with more of the PIVX community (and or other developers) – and now we’re here playing correct the errors and re-attune the narratives that are out in the webs.

There does appear to be some aberrant behaviors in the PIVX network which are being investigated, and as of the time of writing this, there were already some solution sets viable to correct this.  Of note with all of this – I’m not sure this is as much a PIVX specific issue as it appears to be a Proof of Stake consensus issue. The “overlapping” variable here is that 700+ projects have cloned the PIVX codebase at some point in the past.  However, my gut says there is a fundamental issue in the way in which the consensus mechanism rewards which is being “gamed”, and that this is not exclusive to the PIVX network, but rather is in the nature of the Proof of Stake itself. THUS, the continual expert work needed towards a more fair consensus method in Proof of Stake.

Rolling out any network update takes careful testing and verification in PIVX as the developers are committed to putting forth the best possible solution that maintains the integrity and stability of the PIVX network.  Once these are fully validated, then (as is responsible) the PIVX developers will provide the world with the solution sets, which all are then free to use.

Till then, I gladly welcome any dialogue here or in discord, and look forward to seeing what the PIVX developers roll out next.  If history is any indication, PIVX will push something else that “quietly” becomes another part of the backbone of the crypto/blockchain ecosystem – yet another testament to the quality of work being done by those individuals working on PIVX.