Hello, Thank you. I would tend to agree. I've gone back and checked the reverse IP of the connection it's definitely a breech. I've got hidden directories under several places in the filesystem, nothing under /tmp, I don't get how this was done? Am I dealing with a rootkit or not? I ran chkrootkit which reported the suckit rootkit but running rkhunter revealed nothing, and both were fully updated. I've done google searching and apparently that suckit rootkit diagnosis has a history of false positives with chkrootkit. Any help appreciated. Thanks. Dave. On 7/9/14, Scott Granados wrote:

This sounds to me like you got owned. I've noticed that a lot of penetrations like that have hidden directories and or stuff set up under /tmp. Remember that openssl needed to be updated recently as well as anything compiled using it. Not sure in your case whether they were fully successful but if directories appeared I'd tend to think so.

On Jul 9, 2014, at 1:50 AM, David Mehler wrote:

...

Hello Everyone,

I'm running an fc20 Linux linode. I'll admit the past two months I've not been paying as much attention to it as I should, with a bunch of unexpected stresses.

That being said it is fully up to date, I do that daily and it is running a firewall as well as the fail2ban software for automated bots defense.

Tonight I checked my email and noticed two from an account on my system sent to me that is my account. I've got all root email forwarded to my account to read it. Now two things, first this account shouldn't have got any cron reports, second since I use virtual users and they're all in a database that user shouldn't have been able to send email from my server to me.

I checked the account on the server and noticed a hidden directory called .local under which there was a share directory and a system directory under that. Finally there is a broken symbolic link called user which points or pointed I should say to a now nonexistent file and folder which do not exist.

I've got an email out to the user, and I've checked the password file for any accounts with login shells, nothing new has come up.

If this was an attempted root breakin with a rootkit this user is not in th e sudoers file so I think I'm safe there.

I'd like to hear your thoughts on this, and some further steps I can take on this issue to either confirm or refute as to whether I've been breeched.

Thanks. Dave.

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

I would reinstall using known good backups. Trying to clean a server may be a lost cause. Sent from my iPhone

On Jul 9, 2014, at 1:27 PM, David Mehler wrote:

Hello,

Thank you. I would tend to agree. I've gone back and checked the reverse IP of the connection it's definitely a breech. I've got hidden directories under several places in the filesystem, nothing under /tmp, I don't get how this was done? Am I dealing with a rootkit or not?

I ran chkrootkit which reported the suckit rootkit but running rkhunter revealed nothing, and both were fully updated. I've done google searching and apparently that suckit rootkit diagnosis has a history of false positives with chkrootkit.

Any help appreciated.

Thanks. Dave.

...

On 7/9/14, Scott Granados wrote: This sounds to me like you got owned. I've noticed that a lot of penetrations like that have hidden directories and or stuff set up under /tmp. Remember that openssl needed to be updated recently as well as anything compiled using it. Not sure in your case whether they were fully successful but if directories appeared I'd tend to think so.

...

On Jul 9, 2014, at 1:50 AM, David Mehler wrote:

Hello Everyone,

I'm running an fc20 Linux linode. I'll admit the past two months I've not been paying as much attention to it as I should, with a bunch of unexpected stresses.

That being said it is fully up to date, I do that daily and it is running a firewall as well as the fail2ban software for automated bots defense.

Tonight I checked my email and noticed two from an account on my system sent to me that is my account. I've got all root email forwarded to my account to read it. Now two things, first this account shouldn't have got any cron reports, second since I use virtual users and they're all in a database that user shouldn't have been able to send email from my server to me.

I checked the account on the server and noticed a hidden directory called .local under which there was a share directory and a system directory under that. Finally there is a broken symbolic link called user which points or pointed I should say to a now nonexistent file and folder which do not exist.

I've got an email out to the user, and I've checked the password file for any accounts with login shells, nothing new has come up.

If this was an attempted root breakin with a rootkit this user is not in th e sudoers file so I think I'm safe there.

I'd like to hear your thoughts on this, and some further steps I can take on this issue to either confirm or refute as to whether I've been breeched.

Thanks. Dave.

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

Hello, Yes, the connection was from an IP that can't be the users real IP, I have no doubt this account was compromised. I do run ssh and for most accounts they log in via public key authentication, for three users they're in a special group that are allowed passwords. Brute forcing I'm assuming that happened while I was attempting to fix the issue with fail2ban and firewall-cmd don't get me started there. I'm not sure if I am dealing with a rootkit or not, but in the compromised account's home directory I have a hidden folder called .local. Under there there's another folder called share. Under there there's another folder called system and in there there's a symlink called user that points to ../../config/user that directory location does not exist nor does the file, a file check on that user symlink reports it as a broken symlink. I've also run a find on the system I do not have any hidden .config directories. Now all that .local hidden directory stuff it's in the compromised account, or the account I'm thinking as compromised, it's also in two other accounts all allowing password authentication, I've got a hidden .local directory in my web store, which that shouldn't be there at all. The ost disturbing aspect and I have no idea how this was accomplished I have a hidden .local directory same setup as above in root's home directory. As I said chkrootkit reported /sbin/init compromised with the suckit rootkit, but googling reveals that chkrootkit reveals that as a false positive quite a lot. Running rkhunter on the system revealed nothing. Thanks. Dave. On 7/9/14, John G. Heim wrote:

What connection? Are you saying someone logged in from an IP that couldn't possibly be the real user's IP?

That is a good reason to think the user's account was compromised. But that doesn't mean they got root access. This kind of thing happens all the time. If you are running a server with ssh access for lots of accounts, it's not particularly rare for someone to get a password for one of your end users. They can get it by guessing. They guess passwords by geting the user's facebook or twitter password and then trying it on your machine. Or they brute force it by guessing every possible password. If someone's password is 12345, it doesn't take long.

Do you have some reason to think the hacker got root access other than that an end user's account was compromised?

On 07/09/14 13:27, David Mehler wrote:

...

Hello,

Thank you. I would tend to agree. I've gone back and checked the reverse IP of the connection it's definitely a breech. I've got hidden directories under several places in the filesystem, nothing under /tmp, I don't get how this was done? Am I dealing with a rootkit or not?

I ran chkrootkit which reported the suckit rootkit but running rkhunter revealed nothing, and both were fully updated. I've done google searching and apparently that suckit rootkit diagnosis has a history of false positives with chkrootkit.

Any help appreciated.

Thanks. Dave.

On 7/9/14, Scott Granados wrote:

...

This sounds to me like you got owned. I've noticed that a lot of penetrations like that have hidden directories and or stuff set up under /tmp. Remember that openssl needed to be updated recently as well as anything compiled using it. Not sure in your case whether they were fully successful but if directories appeared I'd tend to think so.

On Jul 9, 2014, at 1:50 AM, David Mehler wrote:

...

Hello Everyone,

I'm running an fc20 Linux linode. I'll admit the past two months I've not been paying as much attention to it as I should, with a bunch of unexpected stresses.

That being said it is fully up to date, I do that daily and it is running a firewall as well as the fail2ban software for automated bots defense.

Tonight I checked my email and noticed two from an account on my system sent to me that is my account. I've got all root email forwarded to my account to read it. Now two things, first this account shouldn't have got any cron reports, second since I use virtual users and they're all in a database that user shouldn't have been able to send email from my server to me.

I checked the account on the server and noticed a hidden directory called .local under which there was a share directory and a system directory under that. Finally there is a broken symbolic link called user which points or pointed I should say to a now nonexistent file and folder which do not exist.

I've got an email out to the user, and I've checked the password file for any accounts with login shells, nothing new has come up.

If this was an attempted root breakin with a rootkit this user is not in th e sudoers file so I think I'm safe there.

I'd like to hear your thoughts on this, and some further steps I can take on this issue to either confirm or refute as to whether I've been breeched.

Thanks. Dave.

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

-- --- John G. Heim, 608-263-4189, jheim@math.wisc.edu

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

Hello, Thanks for this. I don't get why the .local directories are not in all my user's home directories but some of them and not sure what this invalid user symlink is. This does make me feel better though. Thanks. Dave. On 7/10/14, John G. Heim wrote:

Having a .local/share is normal. http://askubuntu.com/questions/14535/whats-the-local-folder-for-in-my-home-d...

On 7/9/2014 5:06 PM, David Mehler wrote:

...

Hello,

Yes, the connection was from an IP that can't be the users real IP, I have no doubt this account was compromised. I do run ssh and for most accounts they log in via public key authentication, for three users they're in a special group that are allowed passwords.

Brute forcing I'm assuming that happened while I was attempting to fix the issue with fail2ban and firewall-cmd don't get me started there.

I'm not sure if I am dealing with a rootkit or not, but in the compromised account's home directory I have a hidden folder called .local. Under there there's another folder called share. Under there there's another folder called system and in there there's a symlink called user that points to ../../config/user that directory location does not exist nor does the file, a file check on that user symlink reports it as a broken symlink. I've also run a find on the system I do not have any hidden .config directories.

Now all that .local hidden directory stuff it's in the compromised account, or the account I'm thinking as compromised, it's also in two other accounts all allowing password authentication, I've got a hidden .local directory in my web store, which that shouldn't be there at all. The ost disturbing aspect and I have no idea how this was accomplished I have a hidden .local directory same setup as above in root's home directory.

As I said chkrootkit reported /sbin/init compromised with the suckit rootkit, but googling reveals that chkrootkit reveals that as a false positive quite a lot. Running rkhunter on the system revealed nothing.

Thanks. Dave.

On 7/9/14, John G. Heim wrote:

...

What connection? Are you saying someone logged in from an IP that couldn't possibly be the real user's IP?

That is a good reason to think the user's account was compromised. But that doesn't mean they got root access. This kind of thing happens all the time. If you are running a server with ssh access for lots of accounts, it's not particularly rare for someone to get a password for one of your end users. They can get it by guessing. They guess passwords by geting the user's facebook or twitter password and then trying it on your machine. Or they brute force it by guessing every possible password. If someone's password is 12345, it doesn't take long.

Do you have some reason to think the hacker got root access other than that an end user's account was compromised?

On 07/09/14 13:27, David Mehler wrote:

...

Hello,

Thank you. I would tend to agree. I've gone back and checked the reverse IP of the connection it's definitely a breech. I've got hidden directories under several places in the filesystem, nothing under /tmp, I don't get how this was done? Am I dealing with a rootkit or not?

I ran chkrootkit which reported the suckit rootkit but running rkhunter revealed nothing, and both were fully updated. I've done google searching and apparently that suckit rootkit diagnosis has a history of false positives with chkrootkit.

Any help appreciated.

Thanks. Dave.

On 7/9/14, Scott Granados wrote:

...

This sounds to me like you got owned. I've noticed that a lot of penetrations like that have hidden directories and or stuff set up under /tmp. Remember that openssl needed to be updated recently as well as anything compiled using it. Not sure in your case whether they were fully successful but if directories appeared I'd tend to think so.

On Jul 9, 2014, at 1:50 AM, David Mehler wrote:

...

Hello Everyone,

I'm running an fc20 Linux linode. I'll admit the past two months I've not been paying as much attention to it as I should, with a bunch of unexpected stresses.

That being said it is fully up to date, I do that daily and it is running a firewall as well as the fail2ban software for automated bots defense.

Tonight I checked my email and noticed two from an account on my system sent to me that is my account. I've got all root email forwarded to my account to read it. Now two things, first this account shouldn't have got any cron reports, second since I use virtual users and they're all in a database that user shouldn't have been able to send email from my server to me.

I checked the account on the server and noticed a hidden directory called .local under which there was a share directory and a system directory under that. Finally there is a broken symbolic link called user which points or pointed I should say to a now nonexistent file and folder which do not exist.

I've got an email out to the user, and I've checked the password file for any accounts with login shells, nothing new has come up.

If this was an attempted root breakin with a rootkit this user is not in th e sudoers file so I think I'm safe there.

I'd like to hear your thoughts on this, and some further steps I can take on this issue to either confirm or refute as to whether I've been breeched.

Thanks. Dave.

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

-- --- John G. Heim, 608-263-4189, jheim@math.wisc.edu

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

Hello, Yes, the connection was from an IP that can't be the users real IP, I have no doubt this account was compromised. I do run ssh and for most accounts they log in via public key authentication, for three users they're in a special group that are allowed passwords. Brute forcing I'm assuming that happened while I was attempting to fix the issue with fail2ban and firewall-cmd don't get me started there. I'm not sure if I am dealing with a rootkit or not, but in the compromised account's home directory I have a hidden folder called .local. Under there there's another folder called share. Under there there's another folder called system and in there there's a symlink called user that points to ../../config/user that directory location does not exist nor does the file, a file check on that user symlink reports it as a broken symlink. I've also run a find on the system I do not have any hidden .config directories. Now all that .local hidden directory stuff it's in the compromised account, or the account I'm thinking as compromised, it's also in two other accounts all allowing password authentication, I've got a hidden .local directory in my web store, which that shouldn't be there at all. The ost disturbing aspect and I have no idea how this was accomplished I have a hidden .local directory same setup as above in root's home directory. As I said chkrootkit reported /sbin/init compromised with the suckit rootkit, but googling reveals that chkrootkit reveals that as a false positive quite a lot. Running rkhunter on the system revealed nothing. Thanks. Dave. On 7/9/14, John G. Heim wrote:

What connection? Are you saying someone logged in from an IP that couldn't possibly be the real user's IP?

That is a good reason to think the user's account was compromised. But that doesn't mean they got root access. This kind of thing happens all the time. If you are running a server with ssh access for lots of accounts, it's not particularly rare for someone to get a password for one of your end users. They can get it by guessing. They guess passwords by geting the user's facebook or twitter password and then trying it on your machine. Or they brute force it by guessing every possible password. If someone's password is 12345, it doesn't take long.

Do you have some reason to think the hacker got root access other than that an end user's account was compromised?

On 07/09/14 13:27, David Mehler wrote:

...

Hello,

Thank you. I would tend to agree. I've gone back and checked the reverse IP of the connection it's definitely a breech. I've got hidden directories under several places in the filesystem, nothing under /tmp, I don't get how this was done? Am I dealing with a rootkit or not?

I ran chkrootkit which reported the suckit rootkit but running rkhunter revealed nothing, and both were fully updated. I've done google searching and apparently that suckit rootkit diagnosis has a history of false positives with chkrootkit.

Any help appreciated.

Thanks. Dave.

On 7/9/14, Scott Granados wrote:

...

This sounds to me like you got owned. I've noticed that a lot of penetrations like that have hidden directories and or stuff set up under /tmp. Remember that openssl needed to be updated recently as well as anything compiled using it. Not sure in your case whether they were fully successful but if directories appeared I'd tend to think so.

On Jul 9, 2014, at 1:50 AM, David Mehler wrote:

...

Hello Everyone,

I'm running an fc20 Linux linode. I'll admit the past two months I've not been paying as much attention to it as I should, with a bunch of unexpected stresses.

That being said it is fully up to date, I do that daily and it is running a firewall as well as the fail2ban software for automated bots defense.

Tonight I checked my email and noticed two from an account on my system sent to me that is my account. I've got all root email forwarded to my account to read it. Now two things, first this account shouldn't have got any cron reports, second since I use virtual users and they're all in a database that user shouldn't have been able to send email from my server to me.

I checked the account on the server and noticed a hidden directory called .local under which there was a share directory and a system directory under that. Finally there is a broken symbolic link called user which points or pointed I should say to a now nonexistent file and folder which do not exist.

I've got an email out to the user, and I've checked the password file for any accounts with login shells, nothing new has come up.

If this was an attempted root breakin with a rootkit this user is not in th e sudoers file so I think I'm safe there.

I'd like to hear your thoughts on this, and some further steps I can take on this issue to either confirm or refute as to whether I've been breeched.

Thanks. Dave.

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins

-- --- John G. Heim, 608-263-4189, jheim@math.wisc.edu

_______________________________________________ Blind-sysadmins mailing list Blind-sysadmins@lists.hodgsonfamily.org http://lists.hodgsonfamily.org/listinfo/blind-sysadmins