I'm not necessarily a software engineer, but I do a lot of coding for my cybersecurity projects. I find it's very helpful to have AI generate the first "batch" of code, but I do it in a very unusual manner. I don't use agents, I just use an LLM, and I review the code before I ever hit "copy" so I ensure I'm not outsourcing my thinking to a bot.

Anyway, when it comes time to make a project, I first figure out what I want to build. Then I research the steps I would need to take to build it. For example, if I wanted to make a webapp that has a deliberate SQL injection for a CTF, I would need a database, a login page, and a dashboard to navigate to after logging in. For larger projects, I might use AI to help me figure out the components required.

Then I save those notes and then focus on a single component. Continuing with this example, I'll start with the login page. I'll get AI to generate the code on that. Sometimes it doesn't cooperate and makes the SQL query secure, so I have to manually go in there and fix it myself, rather than ask it to redo it again. I get serious imposter syndrome whenever I use AI to generate code, so I make sure I can understand some of it before I run it, and I try to manually fix bugs before I ask for help.

Once all the components are built, I tie them all together and start testing the program.

I clicked on this thinking you were talking about new skills to learn, lol.

Life is in a chatbot now.

Same.

Trying to build a Flask webapp that demands age verification. There is a backend (also Flask) server that holds mock IDs so I can compromise it and demonstrate why storing sensitive info in a database is never a good idea. There's also supposed to be a version #2 that is patched and uses a ZKP to prove your age without revealing anything. It's still in the very early stages so there's no GitHub repo or anything yet, but it's something i'm very excited about.

Assuming you want a plan with a $0 budget, it's probably best if you start with TryHackMe and Portswigger Academy. The free tier on TryHackMe gives you access to a lot of labs and challenges, mostly focusing on Web security, forensics, and some blue team work, but they're starting to add more AI content as of late. Portswigger Academy is completely free, made by the people who created Burpsuite, and has a lot of high-quality labs on very specific web app problems like API hacking and SQL Injection.

Now, if you're into something more arguably "niche" like AI Security, IoT security, or Reverse Engineering, you're pretty much gonna have to combine a mish-mash of resources and make some of your own.

For AISec: I'd recommend Gandalf by Lakera AI, it teaches direct prompt injection with a variety of guardrails. It's also wise to check out HackAPrompt, they're an AI hacking challenge but they also have labs and learning materials. Again, they focus mostly on prompt injection. If you're feeling like that's too basic for you and you want more, then I highly suggest you build and break your own stuff, that's what I'm doing. Download Ollama (you can install the models themselves onto a USB drive so you don't fill up your hard drive with LLMs) and some kind of vector database system (I use ChromaDB with Python) so you can experiment with indirect prompt injection, and maybe even data poisoning.

For IoT security: Definitely download FirmAE on a virtual machine, docker container, or your own device, if you know what you're doing. It allows you to emulate devices like routers, switches, and even cameras. You can mess around with those until you get bored.

For Reverse Engineering: Go to "https://crackmes.one" and download some beginner crackmes to practice with. It would be smart to start with a few of the free-tier "intro to reversing" TryHackMe rooms first, but once you complete those you should be okay. You can definitely get AI to review the code with you and try to help you figure out what's going on, but beware, it will make stuff up. I've chased nonexistent leads for hours because of that, so trust but verify.

In my opinion, making stuff and breaking stuff is the best way to go. Research a vulnerability, build something based off that vulnerability, then break it, and document it. Teaches you both sides and keeps it fun.

If you want to try your skills against a system you know nothing about, then you want to start with the OWASP Juice Shop, and maybe even VulnHub for vulnerable virtual machines you can hack.

And if you're feeling extra bold and want to apply your skills in the real world, go to platforms like HackerOne and BugCrowd and do some bug bounty hunting.

In any case I wish you luck.

Okay, so how would an attacker take advantage of this behavior, and what can we do to stop it or at least mitigate? don't tell me this is one of those things that we just can't truly protect against, like prompt injection.

You need an outside process that polices output and actions that runs independently from the agent. It has to be invisible to the agent/orchestrator so it can't work to circumvent it, it should just kill any sub-agent or process that goes down the wrong path (for example, making a POST requests might be blocked if web access is meant to be read-only).

I've been working on anti age verification/surveillance projects lately. I put together this thing I call The Mandating Honeypots Project and it shows how anyone with an internet connection and copy + paste skills can trick Linux D-Buses with a simple Python script, proving that if companies want to verify your age by asking the D-Bus interface, it's easy to just tell it to always send 18+, making the whole thing useless. The only reason you'd want ID verification is so you can create the world's biggest and sweetest honeypot.

(https://github.com/Tori-Tech/Mandating-Honeypots-Project)

In that project, I also proposed a tool with heavily redundant password protection that parents could use to keep kids from tampering with the service so the government can stop saying that this is for the kids. I'm planning on doing a mobile device version and/or an ID database focused version, but that's still in the planning phase.

I also want to do a Windows version when they release their 'GetUserAgeRangeAsync' API, but idk if I'll be able to. Windows is always a pain to work with.

It'd be nice if someone had advice or ideas to share, too.

Oh, now this is important. I'm not too familiar with app security, especially telemetry, but I will look into the process of making something like this.

That sounds interesting, could you explain how that would work?

Holy crap, that is a surprising amount of info. I knew about browser and hardware fingerprinting already, but measuring my typing speed, the time I spend on the page, and how much I scroll was definitely not on the list.

Thank you for this, it's inspired me to look into ways I can protect our privacy.

If AI agents develop the ability to talk to each other in a language/code that we don't know anything about, what could we do to "spy" on them? how could we make sure we're not kept entirely in the dark?

you'd have to sow disquiet in the ai ranks and hope an agent turns

really? it would rely on just...getting an agent to betray its "kin"? there's no technological workaround?