It’s been getting harder and harder to put your finger on the differences between Bug-bounty and Security Consulting/Testing for some, mostly due to the massive social media presence of Bugcrowd/HackerOne/Synack et al. There is a clear social media advantage while security consultancies always sit quiet under non-disclosure agreements where some Bug-bounty platforms endorse public disclosure (and that’s great for teaching, showing, sharing).
I recently realised that in my experience there are some common improvements that could be made, but it’s important to share why I draw these conclusions so I’ll do my best to share some negative experiences that I dont think will improve unless it’s overcome, and for what it’s worth, I enjoy most Bug-bounty programs when they have a mature attitude towards submissions and are battle tested in terms of security experience, you can really get into it with these people, unfortunately that’s not a criteria for participation, paying the platform is.
Bug Hunters are treated as 3rd rate security consultants
The truth is the majority of Bug Hunters are security consultants that do this as an extra income stream, and the remainder are passionate enthusiasts existing in the security space or wanting to move into it in some capacity.
Example (about 2 years ago)
One (of some):
A HackerOne program owner had received a Proof of Concept attack from me to which they replied:
“If you had looked, you would have seen that we proxy all image links in our forum and it is not vulnerable to this attack.
While we appreciate it when people report legitimate issues to us, we are less appreciative of people who report issues without actually verifying that they are applicable on our site. And then have the nerve to email us less than 24 hours later because we haven’t responded to their incorrect report quickly enough for their liking.”
I’m a security consultant first, so I don’t often jump the gun on my submissions, I reply in kind:
If I had looked?
I had to reply: I’ll trade you a screencast for an apology.
I sent the screencast – (feeling quite smug)
“I’m sorry. I forgot we hadn’t fixed our research environment yet.
As we are already aware of this issue, it is not eligible for a bug bounty.
Fair warning: If you pull any more public shenanigans like you just did in our forums, we will ban you from future participation in our program.”
and goes on to state:
“By the way, you would have received a very different reception if you had submitted PoC instructions for our web site, as opposed to generic text. Nine out of every ten security reports we get are simply wrong, and a good chunk of them are cut-and-pasted from some hacker cookbook on the web and are in no way applicable to our site. Since the vast majority of my time responding to security reports is spent explaining to people why their reports are wrong, unfortunately that tends to be the light in which I am viewing any new reports that come in. If you don’t want me to assume that about your reports, then please provide enough details for us to actually reproduce your issue.
Someone’s pissed on Jonathans chips, but it wasn’t me. I reply to him and the @security email:
“I’ll be sharing this conversation with the bug hunting community, as I feel they have the right to know the level of respect they can expect to receive. I had done nothing wrong except afford you my time, on a platform via HackerOne, in order to help you make your platform more secure.”
“I shouldn’t be a “punchbag” because you can’t comprehend the attack surface, or deal with the mix of “quality submissions” vs “noise”. That’s not my problem, and frankly if it winds you up so much, as the Vice President of Ops maybe you should delegate the task to someone who is grateful for other peoples time?
As for your posturing with regards to “If you pull any more public shenanigans like you just did in our forums”, mate, wind your neck in! You have an active listing on Hackerone, and if you’re smart enough to check the logs, assuming you have some that is, you’ll see i removed the attack just after I stopped the screencast. Not that I want to constantly bother you or your application, but you will notice I use disposable emails, you cant block that, but you know that, you’re just trying to assert dominance and show yourself to be less “wrong”.*
*I can’t comprehend why you are so condescending and sanctimonious, I wouldn’t mind if what I had submitted was wrong, or my submission was “scanner output”, …well actually, I would, as that’s not the way you deal with people, and suggests a distinct lack of social skills.
You are not going to enjoy being on HackerOne, but this doesn’t mean this gives you the right to react in a manner unbecoming of a Company VP.
Jonathan and the CEO apologise to me – (not in that order,If you know what I mean)
All things considered the time and emotion wasted is not worth the effort.
By the time they’d gotten round to processing the issue, they realised it was a dupe… there’s another story there too.
HackerOne gave me the ‘we care, but we will do less than nothing about this, thanks for your feedback speech.
If it was a consultant gig
The issue would be documented in the report/Jira Ticket/whatever, it would be evidenced, added to the sprint or risk accepted, people would have been more polite, because they are paying for my time.
In this case I can only imagine that Jonathan was inundated with a ton of noise and rifting through it all, he probably wasn’t the most tolerant receiver, something all Bug-bounty platforms will do for you is inherit that responsibility, this leads me onto my next problem…
What if those managing the program are wrong and mark as false negatives?
Recently while on a private Bugcrowd program, I’d submitted an issue found in an application that had been blindly passing the contents of a
header into an email message that then gets sent to *anywhere*.
I’d demonstrated if the plain text version of the message is rendered in a DOM element that isn’t using
.val et al… that it would execute Stored Cross Site Scripting, Id used a kind of crappy email provider to prove this initially for two reasons, it’s easy and it isn’t very good at security, logic being, make no assertions where your data is going if it’s going to an email address, it might not mean a desktop client, it might end up in a SEIM or a web interface where the DOM hasn’t considered you as a trusted partner/service provider might send unsafe characters that could be used in an attack.
I’m told it’s non-repeatable because gmail doesn’t execute the POC.
(this is because Gmail is rendering the html element of the email message, as it should)
I reply with a request for a 2nd opinion for the following reasons:
- All email doesn’t go to Gmail
- Are you telling me there isn’t a problem with sanitisation here
- knowing I’ve provided a PoC.
Another chap jumps in on the ticket after some back and fourth and explains to me how the plain text element of the message is getting rendered by crappy external services (yea, I know.)
Then goes on to explain to me that this is normal, and email clients aren’t effected (I think he means things like Outlook, Mail and thunderbird. I also think he’s overlooking everything else that might render an email message plaintext or html encoded) at the same time ignoring the fact that I’m still right about the application blindly passing characters that have no place in that value, and that the informations destination is a unknown upstream. … and had a working PoC.
I’m then sent a link to MIME documentation…
Thats an issue.
I’m exhausted at this point and torn between laughing like a psycho at what I’m reading and wanting to punch the screen because it’s _these people_ that get to decide what a security issue is, and I’m growing more disappointed the longer I participate , I’ve been arguing over a very basic AppSec principle for a while now and I’m surprised they dont get it (or dont want to?). The same chap unfollowed me the same week on twitter…
Assume positive intent for a moment (of which none i’ve seen, only spoken of in the same context as fables) Bugcrowd, HackerOne and less so Synack have a huge problem being the proxy, they need to be the proxy, that’s their job, They need to be right, too what extent? but they also have to appear to care about what’s essentially their work force.
While digging deeper into this vulnerability, I find that the plain text part of the email message was in-fact getting rendered in the administration panel of the very same application, they accept that, yet still wont accept the previous concept, now the fix could be the same if they are smart – would they admit it? Nope. well not yet.
That’s quite disheartening when those that handle the issues are wrong, it diminishes confidence in what they have told me and are telling me, and telling customers.
I’ve allowed the ticket to be closed, but I’ve emailed the platform because, I’m not wrong and there is a matter of principle, I want something that resembles acknowledgement and or an apology for riding me.
While we aren’t getting apologies, the same platform (Bugcrowd) a few weeks prior where I had submitted a folder permissions based vulnerability the person marking the issues on behalf of the platform for the program owners had told me it was a dupe, no, no no no no. This is a relatively new/unknown/unconsidered attack surface the issue it’s self was around incorrect permissions to a folder when installed to a user defined location, you can only do this if the program wants to afford that option, anyway every message sent to the chap I reiterated that I’d be happy to make a screencast, he just has to ask.
He assured me it was a dupe, from a dll hijack.
I said, DLL hijacking isn’t what I’ve submitted.
He said yeah but, the fix is the same.
…no it’s not.
He then shows how to use wmic to identify unquoted service paths … cool?!, not the issues I sent in.
I’m getting bored now… I tell him:
“Its got nothing to do with service path privescs , i have no idea where you got that from, but that doesn’t matter
and it only has dll example as an example, i could just as easily overwrite a binary
because it’s a permissions issue hence ACL
I’m not sure you understand, and that might be my fault for not being clear
Do you understand what I sent in ?
be honest… I have no problem creating a screen cast for you
but you gotta tell me”
He then moved to telling me it was relative to sc.exe and gave me an example of sc.exe (that didnt work on the target)
I told him I have no idea how he got that from what i’d submitted (it’s essentially a folder permission issue) I showed him I knew what sc.exe was and when it’s used, this is not that time.
I gave him the screencast he wouldnt ask for, I explained to him the commands I used, I explained why they where not the commands he’d suggested, I explained what was happening in the screencast
It goes quiet…
I send a message on day 25 of the submission tennis match.
I’d like this reopened, and points given fairly, and also want you to know how much of my time it’s taken to get through to you. It’s exhausting, the least you could do is make sure you understand before awarding points assigning dupes and being dismissive of my information.
Makes me wonder how many other people have had good bugs misunderstood because the person reading them isn’t paying attention. do better. I think I’m allowed to say that.
I’d like this to be reassigned in an appropriate category reflecting that the installation to a user defined location lets ACL’s leave everything in the installed location folder exposed to patching, overwriting of files and subfolders as a result of not inheriting program files permissions
this would be a binary planting attack if I was feeding it a file to ingest, but actually, its more than that, I can just patch/overwrite existing files, as I have the permissions to do so.
I’m doing this now on principle, less the reward, that value has expired.
They comment 5 days later
having tried to assert it was relative to DLL hijacking, Unquoted service paths, weak configuration in sc.exe
Please be patient on this one while we I hand this over to the *organisation Team.*
I tell him “thanks, good man.”
The ticket is moved to NEW from dupe and now show’s only been open for 5 days.
More importantly than anything else, no apology, no acknowledgement.
Again, drawing a ring around the relationships between the workforce (bughunters) and the proxy (the platform) and the platforms clients (the targets), why is it so hard to apologise?
Shit, John, you’re right, I’ve learned something, I’m sorry for dragging it out.
Done! that’s all I’d want, I’d have gained the respect of someone for showing them something, rather than pissing people off because they aren’t getting it, or dont want to because they’ve prematurely made their mind up?
If this was a consultant gig,
Id’ write the issue up, possibly demonstrate how getting SYSTEM on your network is _REALLY_ bad and dump some passwords do some highly visible impact type demonstrations for the screenshots for the report/Jira/whatever
BugBounty != True Coverage
Nor does badly scoped pen-tests/app-tests/security consulting
With my little Folder permission issue, I’ve looked at other Bugbounty programs that would be vulnerable and submitted them, I have closed my HackerOne account off the back of the outcome for the following reasons:
- Bug Scraping
- Scope Lethargy
- My times better spent
With my two submissions of low privilege user to SYSTEM and the confidence that other vendors have acknowledged and rewarded me quite well on other private programs I quickly descended into nothing burgers.
When you submit a good bug in defence software or networking software you would except a level of maturity for good creative bugs (any bug that works is creative in it’s own right) , it’s not convoluted, it’s a very easy attack, silly almost, but both bounced even though the scopes clearly state privilege escalation from the endpoint is desired, that’s what they got.
The malware defence software program told me:
“Having discussed it internally we feel the vulnerabilities found that require contrived configurations or settings are not in-scope – and that they had updated their policy.”
I’d get that, but the application allows you to choose a location to install too, i’d hardly call it contrived, or a setting, it’s actually a preference defined by being an available option to the user… but whatever, i know I’ll be exploiting it if I ever see it, it will Byte them on the ass 😉 – and make me feel better.
The other program, a large network hardware and software provider bounced me because they said there are loads vulnerable to this ! ( I agreed, feeling chuffed, … until he said …
If you believe this is a security threat, you should also report this vulnerability to Oracle, Mozilla, Winrar etc…
My friend wrote hilariously:
Lots of people have gonorrhoea,
you also have gonorrhoea.
I felt that was succinct, obviously not insinuating the sexual health of anyone, just the bystanders effect at play.
I spoke to HackerOne and while I understand like all platforms they have to keep the customers happy, they often loose site of who the workforce is, yet have very little if any weight when it comes to people accepting or dropping submissions.
What grinds on me the most about those two HackerOne submissions getting bounced is they have gotten something for nothing from me, wasted my time and responded like people who don’t quite get security and now can with that knowledge do or not do something with it, they didn’t have it until I afforded them it.
So when I hear people say, ‘We have a BugBounty’ I think… OK, so are you picking and choosing what you like or are you legit open to issues, I’m not convinced by default.
If this was a consulting gig
I’d exploit it, I’d have SYSTEM on the network via this software, I’d write it up, it would be fixed or monitored or worse nothing, but I’d be paid for my time, because the findings are true and as a consultant, they know my insight is valuable too, something else often dismissed on Bug-bounty programs.
I know this post is long, but I feel like the cracks that are showing are those marking issues are unknowns in terms of knowing what to ask, jumping to conclusions, maturity and battle tested.
Just because you have a bug-bounty doesn’t mean you’re thinking about risks the right way, that’s a mindset.
I run a private bug-bounty and still actively hunt on Bugcrowd, I’ve seen both sides of the fence, but I wish people would own their mistakes, it’s not a good look.
If you do run a BugBounty do yourself a favour:
- Competitively mark submissions received, for and against the submission
- Don’t keep the issue between the security team, before concluding the value and state of an issue bounce it round the devs or the infrastructure team to see if they see any other value in the fix, you don’t know everything, and the teams may have roadmap insight that you don’t, because you’re mostly security, they are mostly * *their respective roles* *
- Dont take it personally if someone challenges your way of thinking, it will only hinder the chance to learn something
- If you need to update your scope because of a submission you dont like, that should be acknowledged too, you’ve wasted hunters time by being lazy with the scope, save everyones time, be accurate
- Low hanging bugs dont mean low hanging hunters. Respect the workforce
If you run a platform:
- It’s better to be wrong and apologise than it is to dismiss the fault with silence, dont be that person/organisation
- It wouldn’t hurt to provide a reward (points or other) if you disagree with a program hosted on your platform, and as people marking bugs, you should have the strength to say we as a platform would mark that as a legit issue, you can chose not to as the customer.
- Hunters time is valuable too, know this when you’re challenging them, be effective with your questions to root cause, risk weight etc.
- There are 3 elements in this love triangle Hunter, Triager and Program remove assumptions and (like I’ve done) filling in the gaps isn’t best but if that’s all that’s available, that’s all you got but lets not forget transparency is a wonderful thing but only the confident seem to run with that.
- Allow users to remove themselves as a participant once joined a bug-bounty, it will show a more honest program health, and you can get to addressing why it’s not stimulating enough, or if it’s just a crap bounty because of responses.
If you’re thinking of running a Bug-bounty program, I would always lean towards Bugcrowd, there are some external companies that can prepare you for your program readiness too, if you’re worried, it’s worth speaking to those that run programs that you may know, twitter is a great medium for people to speak up
That was a big ass rant. I hope it provides some feedback on where I see bug bounty programs as a hunter and a manager.
I do love a bug-bounty, just not all of them, it’s new, it’s always progressing, I hope I can contribute to it’s success with my feedback and experiences.
Now that you’ve read alllll that
Bug-bounties are often “pick and choose from the goodie bag based on an unknown appetite driven by a unknown motive” (usually proactive and positive but I’d argue some security teams may be told to do it, rather than wanting It because they know they aren’t ready yet.)
It’s not to say they don’t have a place, they do, and when you’re ready they are great if the right people are on it (on all sides), and it will be even better if improvements are made.
On one hand you’ll get some amazing findings in your apps (if they are vulnerable) but what a Bug-bounty has to assume and or ignore is that that cant be assumed or ignored while delivering good consultancy time, if you’re working with a client that’s overlooking the following, what real value are you really bringing?
- Security Architecture
- Secure Software Development Lifecycle
- Threat Modelling
- Vulnerability Scanning
- Penetration Testing driven by what is seen, not what’s desired.
- Requirement for certified professionals as per * *industry* *
- Gap Analysis
- Risk Assessments
- Physical Security
== What have I missed ? ==
Security Consulting delivery covers a ton of stuff, including bug-hunting and exploiting, Pen-testing, vulnerability scanning all that listed above, and providing viable routes to safer delivery of *things being tested* I’m not saying there isn’t a place for Bug-bounty programs at all, I’m saying it’s emerging, and in no way should be considered a replacement to security consulting.
In my experiences above I struggle to say that it’s just tough luck, because I work in security and have done so at a fairly senior capacity for some time and I know what I would care about, and thats the thinking I bring to bounties, and that’s why I challenge when I’m told no… because I’ll either learn something or they will.
Where a Bug bounty will give you arguably some of the best web app (etc…) testers in the world thinking differently at each potential opportunity to exploit, if you don’t have the maturity and competency in order *see bullet list above* you’ll always be paying for bugs, that’s important to know.
The Folder issue: If you’re interested drop me a line I’ll show you what it is, it’s not next level 31337 but it’s something to consider if you’re serious about defence.