Running sharepoint on (vm) OBOE. This backs up to THEORY (which is also the VM host). The directory on THEORY is called "theHeapBackup" -- and each backup is yyyy-mmdd-hhmm stamped. Each backup consistes of the SharePoint 2010 content set and the c# source library (for experimental programs like sp and spCmd)
Dependencies/Caveats
Dependent programs are assumed to be available on PATH (and not necessarily in this script's directory.. this is a departure from the accepted standard... sorry). This script uses stamp.exe and roboUpdate.cmd (which has these dependenices: robocopy.exe, stamp.exe and tail.exe). RoboUpdate writes its log file to your %temp% directory, if i recall correctly.
The target machine isn't a member of the domain, so we gotta embed creds (sorry about that). You'll have to visit the value in set myPass=********* and the creds in the net use * %targetRoot% %myPass% /user:THEORY\Nehemiah to get this wet.
The Script
@if not defined DEBUG @echo off
rem backup the heap and the c# source repository 2010-0906
rem put this in \toolkit\tools (uses roboupdate.cmd and stamp.exe
setlocal
set myPath=%~dps0&
set myName=%~n0&
set myPass=*********
for /f "tokens=*" %%m in ('stamp %%w%%a') do set fdr=%%m
for /f "tokens=1*" %%a in ('stamp "%%Y-%%m%%d-%%H%%M %%Y-%%m%%d"') do set dateStamp=%%a&set dirDateStamp=%%b
set stsAdm=C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\BIN\STSADM.EXE&
set targetRoot=\\THEORY\C$&
net use * %targetRoot% %myPass% /user:THEORY\Nehemiah
set targetDir=%targetRoot%\theHeapBackup\%dirDateStamp%&
md %targetDir% >NUL 2>&1
set localSharePointTarget=C:\oboe-%dateStamp%-userSec-ver4&
echo Stsadm to ceate SharePoint backup in %localSharePointTarget%
"%stsadm%" -o export -url http://oboe -fileName %localSharePointTarget% -noFileCompression -includeUserSecurity -versions 4
echo Roboupdate to transfer SharePoint backup to %targetDir%
call roboUpdate.cmd %localSharePointTarget% %targetDir%\sharePoint2010
echo Roboupdate to copy source tree for c#
call roboUpdate.cmd C:\toolKit\src\c# %targetDir%\src\c#
:EXITER
endlocal
exit/b
goto :EOF
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+:+
Monday, September 6, 2010
Saturday, August 21, 2010
Tracking Down Repeated Logon Attempts
Tracking Down Repeated Login Attempts
These things always seems obvious at the end; it was perplexing in the middle; we ignored it at the beginning. The problem smelled a little evil, but there was no evidence of intent. Our initial thought was a wonky scheduled task or some kind of service gone awry.
Discovery
We recently implemented a script to carry out 7-day rolling monitor for failed logon attempts (as recorded across all domain controllers). All counts were in the 1-20 failures per day range. We considered this normal. One admin account (we'll call it _admin) had between 120 and 250 attempts per day. We decided to look into that, thinking some wayward scheduled task somewhere was unable to connect to the network any more.
After some digging we determined that every 10 minutes a logon attempt was being made using the _admin credentials.
We used eventvwr to filter the Security Event Log on eventID 675 -- Kerberos pre-authentication failure. According to 675 the attempt came from one of three machines, each trying about every 10 minutes for about 12 hours in a row. The switch from machine to machine was regular, always in the same order. Since eventID 675 provides client IP address we were able to confine our investigation to those three machines (2 2k3 servers, one xp/pro desktop).
Investigation
On each of the machines we:
• Checked: Scheduled tasks, service credentials, com objects, IIS application pools on each of the servers• Checked: 'findstr' looking for the _admin userid across the file system
• Checked: drive mappings
We decided to focus on the 'current' single offending machine of the three (the one currently generating the 675 events). Using a simple looping script we gathered process list info about 3 times a second over a period of time spanning a login attempt. This produced about 1300 process data files to sift through.
The data files consisted of a single line per active process indicating process name, PID, priority, cmdline and executable path. A second quick script sifted through the 1300 data files looking for differences from file-to-file. This showed that there was no change in the process list before, during or after each logon attempt: It's not a new process; it's an existing process... possibly a service.
Drilling into the unfiltered Security Event Log showed that eventID 680 also occurred at the same time as the 675 pre-authentication errors (eventID 680 is an NTLM authentication event). The internal error code was 0xC000006A - password failure. We also noticed, filtering on 680, that every 680 citing the _admin id pointed to the workstation and not the two servers. I'm not sure why the 675 pre-authentication event cited three machines.
We took the workstation off the network. The 'regular' authentication event did not occur. We put the workstation back on the network. The 'regular' authentication event re-occurred.
Now, with full attention to the workstation, we scanned the service.msc list looking for unusual service entries. There were several including UHPClean, a Microsoft written service, which claimed to be able to clean up local user profiles. We made a quick check on the workstation to verifed that the _admin id had a local profile. We stopped and disabled the UHPClean service. No Change.
Next we fired up perfmon to collect all CPU-related process times (%Priv, %Processor and %User), reasoning that there might be a CPU spike for the offending process at the time of authentication attempt (since we were pretty sure no new process was being created for the occasion). Perfmon indicated a process -- JQS, a Java prefetch procses -- was active around the authentication failure events.
We also noticed a funny process named 'ccmsetup' -- linked to a service with no service description in services.msc.
In spite of the prime directive, we stopped and disabled both of these services. The 'regular' authentication event did not occur. Re-enabled both services. The 'regular' authentication event re-occurred.
After some more iteration we found the ccmsetup service to be the offender. Examining its launch command we saw that it’s related to an SMS package. Since our SMS person is on vacation, further investigation will have to wait. In the mean time, ccmsetup has been stopped and disabled, and the _admin-related Events 675 and 680 have stopped.
Whew!
<technotes>
Tools used in the exercise (free for the asking):
blockfind.exe - like find or findstr, but lets you logically group lines together and flatten a multi-line listing
inNotIn.vbs - uses scripting.dictionary (a hash data structure) to answer the in-not-in question
processList.vbs - any task lister would work; we've had this WMI-based custom lister for a while
processListTimer.cmd - grab process info (see below)
procexp.exe - one of the SysInternals tools
whatsTheDiff.cmd - scan the process list data to see what changed (see below)
Cmdline used to change the multi-line processList.vbs output into 1 line per process:
for /f "tokens=*" %a in ('dir /b *.txt') do @blockfind /file:%a /v processList /oneline /outsep:\0 > bf.%a.bf
Script to gather processList info at about 3 times per second:
processListTimer.cmd
@if not defined DEBUG @echo off
setlocal
set myPath=%~dps0&
set myName=%~n0&
echo will stop when I find %myName%.stop
:LoopTop
if exist %myName%.stop goto :EXITER
set microTime=%time%&
set microTime=%microTime::=.%
set fileName=pl-%microTime%-processList-%computerName%.txt
echo %fileName%
cscript //nologo %myPath%processList.vbs . >> %fileName%
goto :LoopTop
:EXITER
del %myName%.stop >NUL 2>&1
endlocal
exit/b
goto :EOF
whatsTheDiff.cmd
@if not defined DEBUG @echo off
setlocal
set myName=%~n0&
set myPath=%~dps0&
set old=&
set new=&
for /f "tokens=*" %%a in ('dir /b *.bf') do call :Diff %%a
:EXITER
endlocal
exit/b
goto :EOF
:Diff
set new=%~1&
if defined old innotin %old% %new% /notinin
set old=%new%
goto :EOF
Other refs...
http://www.windowsecurity.com/articles/Kerberos-Authentication-Events.html
Audit Account Logon Events at http://technet.microsoft.com/en-us/library/bb742435.aspx#EDAA
Goo http://www.google.com/search?q=event+680+audit+failure+code
</technotes>
Script to compare each of the 1300 data files for differences:
These things always seems obvious at the end; it was perplexing in the middle; we ignored it at the beginning. The problem smelled a little evil, but there was no evidence of intent. Our initial thought was a wonky scheduled task or some kind of service gone awry.
Discovery
We recently implemented a script to carry out 7-day rolling monitor for failed logon attempts (as recorded across all domain controllers). All counts were in the 1-20 failures per day range. We considered this normal. One admin account (we'll call it _admin) had between 120 and 250 attempts per day. We decided to look into that, thinking some wayward scheduled task somewhere was unable to connect to the network any more.
After some digging we determined that every 10 minutes a logon attempt was being made using the _admin credentials.
We used eventvwr to filter the Security Event Log on eventID 675 -- Kerberos pre-authentication failure. According to 675 the attempt came from one of three machines, each trying about every 10 minutes for about 12 hours in a row. The switch from machine to machine was regular, always in the same order. Since eventID 675 provides client IP address we were able to confine our investigation to those three machines (2 2k3 servers, one xp/pro desktop).
Investigation
On each of the machines we:
• Checked: Scheduled tasks, service credentials, com objects, IIS application pools on each of the servers• Checked: 'findstr' looking for the _admin userid across the file system
• Checked: drive mappings
We decided to focus on the 'current' single offending machine of the three (the one currently generating the 675 events). Using a simple looping script we gathered process list info about 3 times a second over a period of time spanning a login attempt. This produced about 1300 process data files to sift through.
The data files consisted of a single line per active process indicating process name, PID, priority, cmdline and executable path. A second quick script sifted through the 1300 data files looking for differences from file-to-file. This showed that there was no change in the process list before, during or after each logon attempt: It's not a new process; it's an existing process... possibly a service.
Drilling into the unfiltered Security Event Log showed that eventID 680 also occurred at the same time as the 675 pre-authentication errors (eventID 680 is an NTLM authentication event). The internal error code was 0xC000006A - password failure. We also noticed, filtering on 680, that every 680 citing the _admin id pointed to the workstation and not the two servers. I'm not sure why the 675 pre-authentication event cited three machines.
We took the workstation off the network. The 'regular' authentication event did not occur. We put the workstation back on the network. The 'regular' authentication event re-occurred.
Now, with full attention to the workstation, we scanned the service.msc list looking for unusual service entries. There were several including UHPClean, a Microsoft written service, which claimed to be able to clean up local user profiles. We made a quick check on the workstation to verifed that the _admin id had a local profile. We stopped and disabled the UHPClean service. No Change.
Next we fired up perfmon to collect all CPU-related process times (%Priv, %Processor and %User), reasoning that there might be a CPU spike for the offending process at the time of authentication attempt (since we were pretty sure no new process was being created for the occasion). Perfmon indicated a process -- JQS, a Java prefetch procses -- was active around the authentication failure events.
We also noticed a funny process named 'ccmsetup' -- linked to a service with no service description in services.msc.
In spite of the prime directive, we stopped and disabled both of these services. The 'regular' authentication event did not occur. Re-enabled both services. The 'regular' authentication event re-occurred.
After some more iteration we found the ccmsetup service to be the offender. Examining its launch command we saw that it’s related to an SMS package. Since our SMS person is on vacation, further investigation will have to wait. In the mean time, ccmsetup has been stopped and disabled, and the _admin-related Events 675 and 680 have stopped.
Whew!
<technotes>
Tools used in the exercise (free for the asking):
blockfind.exe - like find or findstr, but lets you logically group lines together and flatten a multi-line listing
inNotIn.vbs - uses scripting.dictionary (a hash data structure) to answer the in-not-in question
processList.vbs - any task lister would work; we've had this WMI-based custom lister for a while
processListTimer.cmd - grab process info (see below)
procexp.exe - one of the SysInternals tools
whatsTheDiff.cmd - scan the process list data to see what changed (see below)
Cmdline used to change the multi-line processList.vbs output into 1 line per process:
for /f "tokens=*" %a in ('dir /b *.txt') do @blockfind /file:%a /v processList /oneline /outsep:\0 > bf.%a.bf
Script to gather processList info at about 3 times per second:
processListTimer.cmd
@if not defined DEBUG @echo off
setlocal
set myPath=%~dps0&
set myName=%~n0&
echo will stop when I find %myName%.stop
:LoopTop
if exist %myName%.stop goto :EXITER
set microTime=%time%&
set microTime=%microTime::=.%
set fileName=pl-%microTime%-processList-%computerName%.txt
echo %fileName%
cscript //nologo %myPath%processList.vbs . >> %fileName%
goto :LoopTop
:EXITER
del %myName%.stop >NUL 2>&1
endlocal
exit/b
goto :EOF
whatsTheDiff.cmd
@if not defined DEBUG @echo off
setlocal
set myName=%~n0&
set myPath=%~dps0&
set old=&
set new=&
for /f "tokens=*" %%a in ('dir /b *.bf') do call :Diff %%a
:EXITER
endlocal
exit/b
goto :EOF
:Diff
set new=%~1&
if defined old innotin %old% %new% /notinin
set old=%new%
goto :EOF
Other refs...
http://www.windowsecurity.com/articles/Kerberos-Authentication-Events.html
Audit Account Logon Events at http://technet.microsoft.com/en-us/library/bb742435.aspx#EDAA
Goo http://www.google.com/search?q=event+680+audit+failure+code
</technotes>
Script to compare each of the 1300 data files for differences:
Subscribe to:
Posts (Atom)