WordPress core is the code that powers WordPress itself. It is what you get when downloading WordPress from wordpress.org, minus the themes and plugins.
At the most basic, the WordPress core loading follows this pattern:
Load MU plugins
Load activated plugins
Run main query
Administration and AJAX requests follow a similar but lighter process. This diagram covers the specifics:
Many plugin and theme developers attempt to unregister the jQuery that comes with core, and add their own copy, normally the jQuery on the Google CDN. Do not do this, it can cause compatability issues.
Instead use the copy of jQuery that comes with WordPress and aim for the version used in the latest WordPress when testing. This ensures maximum compatability across plugins.
It's tempting to modify parts of Core to remove or add things, but this must never be done. When WordPress updates, all your changes will be lost.
Instead, use Hooks/Actions and Filters to modify Core behaviour.