I setup most of my own guitars about once every 1-1.5 years, or on an as needed basis. This is more of a diary post to record what my and measurements are at this current point in time. This post will deal with Strat style guitars, but some of these measurements apply across all my guitars.
I am no luthier, there may be misinformation in this post - as I said this is more of a post for my own reference, as I have changed these steps countless times over the years whenever I find a better way of doing things.
This also assumes the frets have been leveled and crowned. I level my own frets (most of the time) and that is a procedure that is separate to this one.
General Goal(s):
Get an action of 1.5mm to 1.75mm (treble to bass strings)
Neck relief of between 0.010" to 0.013" at 7th fret (thickness of first or second string)
Ability to bend strings up to 1.5 steps (3 frets)
Stable bridge that returns to pitch (within reason)
Max string action of 0.20" at first fret (nut action)
Intonation within reason (usually plus-minus 6-7 cents from target note)
Pickup volumes balanced between low and high E and between each pickup
PS. most of these apply for the Tele setups I do, excepting the bridge vibrato stuff.
Step 1: Straighten neck
Using a capo on the first fret, and fretting the last fret (or use something like an Allen key to depress the B string in between the 2 adjacent strings - see figure A) check how straight the neck is.
Measure the relief on the 7th fret in playing position. I do this a number of ways, depending on what I have on hand:
Feeler gauge
Thickness of B or E strings as a reference - you can use the reflection of the string on the fret as a guide, or the cutoff of one of these strings
Target a neck relief value of 0.010" up to 0.013" at the 7th fret, in playing position.
Also, make sure there have been no recent temperature changes. If there have been, wait for a few hours before performing any changes).
Step 2: String Action
At this stage, neck has proper relief. I've also assumed that the nut height is CLOSE to where it needs to be. If it is a new nut, I might rough out this step and move up to step 3, then back down to step 2...keep iterating until I am happy with the adjustments.
I measure the action at the 12th fret, with strings open. This does NOT take the nut height out of the equation (hence my earlier statement). All measurements here are taken in playing position.
I set the strings to be 1.5mm at the high E and 1.75mm at the low E. All the other strings will obviously be between those min and max values.
I then check any of the strings fret out when fretted or bent up to 1.5 steps. If they do fret out, I increase the action to accommodate. Note that this needs to be rechecked once new strings are put on the guitar, readjusted if necessary.
Step 3: Nut height adjustment
Nut slots shouldn't change much year to year unless they have been worn down or there have been changes in string gauge. They can also change when fret levelling has occurred because the fret itself has decreased in height.
I generally aim for a reading of 0.020" to 0.018" from bass strings to treble strings.
I find it easier to use the String Action Gauge/Ruler to get it into the right ballpark, and then use the automotive feeler gauges from there (only because my string action gauge doesn't have a 0.020 or 0.018 measurement.
Readings are taken with the guitar resting on a flat surface.
Nut files are used if the slots are not deep enough. Nut is reshaped as necessary, usually I leave about half of the wound strings sitting clear of the the nut.
When cutting the nut slot, I ensure that the angle is the same as the angle of the strings. I find that it might help using some graphite in the slot to help see if you're "cutting" all the way into the front (fretboard side) of the nut slot. The guitar string should be resting on the whole of the nut, i.e. the string leaves the nut at the start of the fretboard and not the middle of the nut slot). This is necessary to ensure it can be intonated properly.
Step 4: Remove Strings, Clean up Frets and Fretboard, Restring
Remove and restring should be self explanatory.
For cleaning up the frets I use either micro mesh or a fine nail polish file. I have also used a dremel with a felt wheel and some compound. Tape up the fretboard as necessary.
Step 5: Stretch out the strings while checking for notes that fret out
Stretch 'em out but pulling the strings with my right hand while fretting up the fretboard. Retune and reiterate.
I will also stretch the strings by making big bends (especially the unwound strings). If they fret out then adjust the action as per Step 2.
Step 6: Pickup height
Only do this if I'm not happy with the way it sounds (too trebly or not enough output, or if there have been new pickups put in), but generally skip this step.
What I look for:
I generally set single coils to be quite far from the strings, definitely much lower than the "stock" setups. I do this because I find it helps with the sustain and avoids any warbling with the notes
I usually end up angling the treble side closer to the strings after the pickups have been lowered. I just generally aim to get the low E and the high E to be roughly the same volume.
I usually set the neck pickup first, and then go from there. There's no reason for this, except that I have to start somewhere. I will balance the other pickups with the neck pickup.
I generally use my ears and don't really take measurements too seriously. I might use a levels meter in Amplitube to assist in adjusting pickup height, and I might do this exercise across 2-3 different sessions to give my ears a rest.
Step 7: Intonation
Usually I'll use the most accurate tuner I have (which, at this moment is the tuner in Amplitube). Detune and adjust saddles backwards or forwards as necessary, then tune up again. Generally aim for within 6 or 7 cents of the target note, by doing a fretted note vs open note. I will also compare the natural harmonic on the 19th fret with the fretted note (also 19th fret) by ear. They should be the same pitch.
Then I try and play notes up and down the neck and take note of how much the note is "off" on each area of the neck. Again, anything within a few cents I will ignore...but if there are major discrepancies it will warrant further investigation.
Parting words
That's pretty much how I do my strat and tele setups. I take my time with each step and there are times where I may have to step backwards, then step forward in order of the steps again. It's a system, and many times changes in one area will affect 2 other areas. That is why the order of steps is important, but it is also why you may have to step back a few steps and redo something if you're not getting the result desired.