Did the Buddha ever define what he meant by “self”?

Anattalakkhaṇasutta is a short sutta for "self" explanation. 

Brahmajālasutta is a long sutta. 

All abhidhamma, especially paṭṭhāna, are "non self" explanation. This is the longest explanation. 

If you can't understand "self" by all of these sutta, it means your pali skill is lacking.

There are only two realities: saṅkhāra and asaṅkhāra; All others are paññatti

There are only two realities: saṅkhāra (dependent origination such as aggregate) and asaṅkhāra (no dependent origination such as nibbāna). All others are paññatti (concept;imagine;not reality) because saṅgkhāra must have tree sub-characteristics and asaṅkhāra must have not tree sub-characteristics. But paññatti has nothing, it can't arise like saṅkhāra and can't cease saṅkhāra like nibbāna.
By VN Parivāra of upāli-etadagga:

Aniccā sabbe saṅkhārā, dukkhānattā ca saṅkhatā; Nibbānañceva paññatti, anattā iti nicchayā.
Saṅkhāra, which means saṅkhata(effect-saṅkhāra)*, is anicca&dukkha&anattā; But only nibbāna&paññatti which are explained as only anattā.
(*Saṅkhāra, which means cause-saṅkhāra, use in more fixed meaning word, such as saṅkhāra-khandha, saṅkhāra-paṭiccasamuppāda. You can notice it in each context, because it is already appear clearly.)

And by KN Paṭisambhidāmagga of sāriputta-mahāsāvaka:

sabbe saṅkhārā aniccā sabbe saṅkhārā dukkhā sabbe dhammā anattāti
All saṅkhāra are anicca, all saṅkhāra are dukkha, and all dhammā are anattā.

There is no saṅkhāra which is not aggregates (Khandhavinimuttako hi saṅkhāro nāma natthi.)
Below, the second group is the conclusion of the first group, Sutta. Ma. Mū. Cūḷasaccakasuttaṃ:

"Aggivessana, 2 the Blessed One disciplines his disciples in this way; this part of the Blessed One's instruction is generally presented to his disciples: ('Form is inconstant. Feeling is inconstant. Perception is inconstant. Fabrications are inconstant. Consciousness is inconstant.) Form is not-self. Feeling is not-self. Perception is not-self. Fabrications are not-self. Consciousness is not-self. (All fabrications are inconstant.) All phenomena are not-self.' This, Aggivessana, is the way in which the Blessed One disciplines his disciples; this part of the Blessed One's instruction is generally presented to his disciples."

However, "All phenomena are not-self" in above context include more than "Form is inconstant", because it include more two, nibbāna and paññatti, as I have described above.
Abhidhamma and atthakathā have this same explanation as well.

Jhāna meditation defination

I am so sorry for the long answer. It's required all of this knowledge to understand the meditation's idea.

What is jhāna?

Jhāna (concentration meditation) is the meditation method to cease the desire (chanda-rāga) of the world (loka) by changing mind's attention to the opposite object of the world.

What is the world?

Living being (satta) is the world (loka), 5 clinging-aggregates (upādāna-khandha[1][2]).

Loka means breakable thing. And what we can notice 3 characteristics must be break, breakable. What is unstable (anicca), stressful (dukkha), and non-self (anattā) is what we can notice it's instability-characteristic (anicca-lakkhaṇa), e.g. it's arising and vanishing, stress-characteristic, e.g. it's stressing to arise and to vanish in every millisecond, and uncontrollable characteristic, e.g. it's variant variables.

Each aggregate has 3 characteristics, so every aggregate (pañca-khandha) is breakable, and it is called Loka. There are 2 kinds of breakable aggregates, lokiya (with clinging) and lokuttara (no clinging). However the practitioner should comprehend only clinging aggregates (upādāna-khandha) to meditate insight meditation, Buddha called clinging aggregates (upādāna-khandha), in first noble truth, not only aggregates (khandha).

The genius people can notice 3 characteristics of every living being, so they try to cease all possible cases of living beings' rebirth by the meditation.

There are many method of the meditation of those genius people which doing the difference methods and each method causes the difference results, some is not able to pause any world, some can pause 5 string world (kāma loka), some can pause form concept world (rūpa loka), some can cease all worlds (kāma loka/rūpa loka/arūpa loka).

How many kinds of jhāna?

There are 3 kinds of jhāna:

1. Form jhāna (rūpa jhāna) - meditation of changing mind's attention form five strings world (kāma-loka) to form's concept world (rūpa-paññatti-loka) to disable 5 sense attachments arising (kāmachandarāga). This meditation's step, 1st-4th jhāna, pause attachment of five strings and mental factors which can drop the quality of concentration down to five strings world. The attained people are going to reborn in the world without five string attachment, but it still has forms, such as eyes to watch the form to meditate rūpa-jhāna.
2. Formless jhāna (arūpa jhāna) - meditation of changing mind's attention form form's concept world to formless world (arūpa-loka) to disable all form arising, which make rūpa jhāna&world still nearby five strings attachment. This meditation change the object to detach from mind and mental factors (arūpa-loka) as much as they can. However, it's useless, the number of mental factors still be the same, 30 factors, from the 1st formless jhāna to the 4th formless jhāna. The attained people are going to reborn in the world without form, but it still have mind and mental factors (nāma-khandha).
3. Three characteristic jhāna (lakkhaṇa-jhāna) - vipassanā; insight meditation. Meditation of changing mind's attention form all worlds to nibbāna (cessation of world). The attained magga people do not reborn any more in maximum 7 rebirths.

Jhāna bases vipassanā, especially for the practitioner who attach five strings strongly (taṇhā-carita).

How to disable 5 sense attachment?

Keeping the sixth sense mind on concept object (paññatti;paṭibhāga-nimitta) by practice to change the the sixth sense memory of the five strings (kāmaguṇa) to concept object (paññatti;paṭibhāga-nimitta) then meditate that practice to go on for a long period, a hour or a day.

How to disable all form arising?

Keeping the sixth sense mind on the object which has no any relation with form (arūpa) by practice to change the the sixth sense memory of form concept (rūpa-paññatti;paṭibhāga-nimitta) to formless object (arūpa) then meditate that practice to go on for a long period, a hour or a day.

What is the problem?

The jhāna practitioner still keep the self attachment of their meditation mind&mind factors as "it's mine. It's me, not the other. It's myself, so I can control it". This is the reason why we need Buddha.

---

See Ma.Mu. Cūḷasīhanādasutta, Mahādukkhakkhandhasutta, the path of purification in pathavīkasiṇa chapter and aruppā chapter, and this article.

What is the meaning behind saṅghānussati chant?

For the meaning of saṅghānussati in advance, see Part II—Concentration (Samádhi) CH. VII SIX RECOLLECTIONS Recollection of the Saṅgha page 215.

We should be used with the tree refuges according to Sutta. Ma. U. Sevitabbāsevitabbasuttaṃ:

It was said, "Sāriputta, I say, persons are also twofold those that should be used and not used" On account of what was it said by the Blessed One? Venerable sir, when using certain persons demerit increases and merit decreases, such persons should not be used, when using certain persons demerit decreases and merit increases, such persons should be used. It was said, "Sāriputta, persons too are twofold, those that should be used and not used. " It was said on account of this.

Why? Sutta. Khu. Jā.(1) Vīsatinipātajātakaṃ Sattigumbajātakaṃ:

“To whomsoever, good or bad, a man shall honour pay, Vicious or virtuous, that man holds him beneath his sway.

“Like as the comrade one admires, like as the chosen friend, Such will become the man who keeps beside him, in the end.

“Friendship makes like, and touch by touch infects, you’ll find it true: Poison the arrow, and ere long the quiver’s poisoned too.

“The wise eschews bad company, for fear of staining touch: Wrap rotten fish in grass, you’ll find the grass stinks just as much. And they who keep fool’s company themselves will soon be such.

“Sweet frankincense wrap in a leaf, the leaf will smell as sweet. So they themselves will soon grow wise, that sit at wise men’s feet.

“By this similitude the wise should his own profit know, Let him eschew bad company and with the righteous go: Heaven waits the righteous, but the bad are doomed to hell below.”

Shyness is Coward; prompted.

Shyness is Coward; prompted.
In abhidhamma, minds and mental factor of sasaṅkhārika (prompted) mind is coward.

What is sasaṅkhārika (prompted)? (src)

12. Sankhārika-
    This is purely a technical term used in a specific sense in the Abhidhamma. It is formed of 'sam', well and Ö 'kar', to do, to prepare, to accomplish. Literally, it means accomplishing, preparing, arranging.
    Like dhamma, sankhāra also is a multi-significant term. Its precise meaning is to be understood according to the context.
    When used as one of the five 'aggregates' (pañcakkhandha), it refers to all the mental states, except vedanā and saññā. In the paticca-samuppāda it is applied to all moral and immoral activities, good and bad thoughts. When sankhāra is used to signify that which is subject to change, sorrow, etc., it is invariably applied to all conditioned things.
    In this particular instance the term is used with 'sa' = co-; and a = un, Sa-sankhārika (lit., with effort) is that which is prompted, instigated, or induced by oneself or by another. 'Asankhārika' (lit., without effort) is that which is thus unaffected, but done spontaneously.
    If, for instance, one does an act, induced by another, or after much deliberation or premeditation on one's part, then it is sa-sankhārika. If, on the contrary, one does it instantly without any external or internal inducement, or any premeditation, then it is asankhārika.

How can the practitioner comprehends shyness?

the practitioner must comprehends below mind, it's mental factors, and the matters born of those mind&mental factors to comprehend shyness:

4 prompted attaching mind (src)

Attaching consciousness, prompted, accompanied by pleasure, connected with wrong view,
Attaching consciousness, prompted, accompanied by pleasure, disconnected with wrong view
Attaching consciousness, prompted, accompanied by indifference, connected with wrong view
Attaching consciousness, prompted, accompanied by indifference, disconnected with wrong view

Mental factors: Twenty-two factors enter into combination - namely, thirteen aññasamānas, and 3 greed mental factors, 2 boring mental factors, and 4 ignorant mental factors (13 + 3 + 2 + 4 = 22)

1 prompted rudeness mind (src)

Rudeness consciousness, prompted, accompanied by displeasure, connected with ill-will

Mental factors: Twenty-two factors enter into combination - namely, twelve aññasamānas, and 4 angry mental factors, 2 boring mental factors, and 4 ignorant mental factors (12 + 4 + 2 + 4 = 22)

12 prompted wholesome mind (src)

Beautiful consciousness, prompted, accompanied by pleasure, associated with knowledge. x3 (wholesome, resultant, kiriyā)
Beautiful consciousness, prompted, accompanied by pleasure, dissociated with knowledge, x3
Beautiful consciousness, prompted, accompanied by indifference, associated with knowledge,x3
Beautiful consciousness, prompted, accompanied by indifference, dissociated with knowledge.x3

Mental factors: thirty-six factors enter into combination - namely, thirteen aññasamānas, and twenty-five beautiful mental factors (13 + 25 = 38)

Anupubbīkathā is an over long sutta

Ugga-richman, lay, said in Sutta. Aṅ. (4): aṭṭhaka-nipātā:

With confident heart I paid homage to the Buddha. The Buddha taught me step by step (anupubbikakathā), with a talk on giving, ethical conduct, and heaven. He explained the drawbacks of sensual pleasures, so sordid and corrupt, and the benefit of renunciation. And when he knew that my mind was ready, pliable, rid of hindrances, joyful, and confident he explained the special teaching of the Buddhas: suffering, its origin, its cessation, and the path. Just as a clean cloth rid of stains would properly absorb dye, in that very seat the stainless, immaculate vision of the Dhamma arose in me: ‘Everything(suffering) which is arise by the origins(samudaya), has an end.’ I saw, attained, understood, and fathomed the Dhamma. I went beyond doubt, got rid of indecision, and became self-assured and independent of others regarding the Teacher’s instructions. Right there I went for refuge to the Buddha, his teaching, and the Saṅgha. And I undertook the five training rules with celibacy as the fifth. This is the second incredible and amazing quality found in me.

After listened anupubbīkathā, some listener can enlighten "dhammacakkhuṃ", i.e. from DN Sāmaññaphalasuttaṃ:

So King Ajatasattu, delighting and rejoicing in the Blessed One's words, rose from his seat, bowed down to him, and — after circumambulating him — left. Not long after King Ajatasattu had left, the Blessed One addressed the monks: "The king is wounded, monks. The king is incapacitated. Had he not killed his father — that righteous man, that righteous king — the dustless, stainless Dhamma eye would have arisen to him as he sat in this very seat."

You can search more result by yourself with word "dhammacakkhu", "anupubbika" and "anupuppīka".

There are many place like that in tipitaka, so Buddha exactly taught four noble truth to the lay. The required questions are:

1. What is anupubbīkathā?
2. Why Buddha often taught anupubbīkathā to a lay?
3. Why the first saṅgāyanā monks have to separate anupubbīkathā as may partitions?
4. Why some people think Buddha never taught meditation to a lay?

1. What is anupubbīkathā?

1. Anupubbīkatha, in pāli canons, is the sequence teaching, story, or history. See, each searching results from tipitaka and atthakathā.

2. Anupubbīkathā, in tipitaka context, the single time teaching of entire instructional sequence. It is the very long sutta which describing about buddhist practitioner's procedure, such as single time teaching of KN Khu Nidhikaṇḍaṃ (with some commentary) + AN Uposathasuttaṃ (with some commentary) + DN Sāmaññaphalasuttaṃ (with some commentary). You can see that is over long to orally recite.

2. Why Buddha often taught anupubbīkathā to lay?

Buddha taught anupubbikathā to everyone, not only lay, because every practitioner must practice step by step. No one can practice without sequence like a fog. See, MN Majjhimapaṇṇāsaka Kīṭāgirisutta Number 238.

However for a newbie lay who is genius, buddha taught him from the beginning step to the end in single time, such as single time teaching of KN Khu Nidhikaṇḍaṃ (with some commentary) + AN Uposathasuttaṃ (with some commentary) + DN Sāmaññaphalasuttaṃ (with some commentary).

You can see that is over long to orally recite.

That's why we can't find the single sutta of anupubbīkathā, in tipitaka context, or even in atthakathā as well. So, in atthakathā comment of anupubbīkathā:

Anupubbīkathā is sequence of teaching and commentary of dāna subjects then sīla subjects then heaven subjects then practitioner's procedure.

But the buddha&teacher taught a shorter and more complex sutta to the insider lay man&lay woman, upāsaka and upāsikā, because they have enough ability and knowledge to learn the higher level, e.g. AN Dānasutta. You can see that sutta is shorter and more complex, included only charity and the Path. It is anupubbīkathā, but it is not the meaning of "anupubbīkathā" word in the context like Sutta. Aṅ. (4): aṭṭhaka-nipātā.

3. Why the first saṅgāyanā monks have to separate anupubbīkathā as may partitions?

The first saṅgāyanā monks chose "anupubbikathā" words to avoid the over long of sutta. Because most of lay at that time don't have much knowledge for an enlightenment, buddha have to taught them very long sutta (maybe more that DN) and mix many contents to let them enlighten as sotāpanna, or trust to dhamma. So, if the first saṅgāyanā monks didn't separate it to be many smaller sutta, it will be over long and unable to orally reciting study.

The evidence from netti vicayahārasampāta:

587. Herein, the Lord Buddha advises one of keen faculties with advice in brief; the Lord Buddha advises one of medium faculties with advice in brief and detail; the Lord Buddha advises one of blunt faculties with advice in detail.

4. Why some people think Buddha never taught meditation to a lay?

Because they are misunderstanding of anupubbikathā. And they don't know how to read teaching as four noble truth, because they never memorize netti, or never attained any professional knowledge, ñāṇa, from tipitaka-memorizer&jhānalābhī school, such as Pa-Auk.

Every noble one must enlightened four noble truth, but it is not every ordinary people can discover the noble truth from tipitaka. People who can understand the noble truth from tipitaka must can deconflict every uncleared word of tipitaka. This is unable for people who can't memorize tipitaka pali and netti, because it needs nirutti-paṭisambhidā to deconflict the whole tipitaka.

For the example from VN Mahāvagga by assachi arahanta to sāriputta, who enlightened as sotāpanna:

Ye dhammā hetupabhavā (dukkha) tesaṃ hetuṃ (samudaya) tathāgato āha
tesañca yo nirodho (nirodha&magga) evaṃvādī mahāsamaṇo.

‘Those things(suffering) which proceed from a cause, of these the Truthfinder has told the cause(origin), And that which is their stopping (cessation&path)— the great recluse has such a doctrine.’”

Or in Sāmaññaphalasuttaṃ, which king Ajātasattu almost enlightened as sotāpanna, it is included vipassanāñaṇa part (brief dukkha), pubbenivāsānussatiñāṇa+cutūpapātañāṇa (extended dukkha), āsavakkhayañāṇa (samudaya+nirodha), and all practitioner's procedures in this sutta are magga.

All above is the example of how to read the noble truth in tipitaka.